5

fcking dropbox. it seems that it doesn't work well with accounts which have over 300,000 files. In my case I have around 1,5 million files and total size is 500gb. Problem is that I am not able to sync them to my local machine. Every time I try to do that dropbox is stuck at "starting" and it's usage builds up till 3.5gb ram and then dropbox crashes.

What should I do if I want to sync everything in dropbox with my local machine in order to go through my old stuff and delete everything I don't need? I could do selective sync and go by chunks of 300k files, but it sounds like a pain in the ass since distribution of files across hundreds of folders is not even.

Maybe there is some better cloud service which deals better with large amounts of files? Maybe I could move all of my files from dropbox to that other cloud service and then sync it on my local machine properly?

Comments
  • 2
    I have up on Dropbox as soon as they said they won't support ext4 filesystems...

    I've just split up a lot of stuff and migrated to private github repo's, been smooth sailing since and have access to more than 2 fucking percent of my cpu
  • 0
    @filthyranter how well do they deal with large amounts of files? Quick google search reveals that they have probs with syncing at 500k files https://help.nextcloud.com/t/...
  • 0
    @filthyranter Im sure dropbox has good servers as well. My problem seeems to be software client related. Client just crashes at syncing.
  • 0
    @filthyranter how many are we talking about? At least 1 mil?
  • 0
    @filthyranter Which provider are you using for hosting nextcloud?
  • 0
    @filthyranter I need more details on your ovh plan. I am not able to find anything with more than 100-200gb storage. Need at least 1-2tb :/
  • 0
    @filthyranter Actually first you said ovh vps with storage, now you're saying some shady remote storage with some shady deals. Anyways thanks for the recommendation man.
  • 0
    Update on the situation: Managed to do syncing via rsync while logged in via dropbox api. However I am limited to around 7 files per second :) So yeah it will take me around 60-70 hours to retrieve 1.5 million of small files :) Moving to my own cloud after this for sure
  • 0
    @filthyranter I bought a kimsufi.com vps from https://kimsufi.com/en/servers.xml/ the plan KS-3. I also installed nextcloud on it by following tutorial from https://howtoforge.com/tutorial/...

    Syncing dropbox with nextcloud via rsync still limits me to 5-6 files per second (I guess this is limited by dropbox API).

    However I noticed that syncing from my nextcloud instance to my local pc is no better. Syncing 3000 files under 5kb takes around 5 minutes. Is it normal or should I tweak my config, remove apache2, use nginx and try other something else?
Add Comment