Learn More
Resend Email
0
zemaitis
5y

Need an advice on syncing large amounts of small files. For starters I have 2 million of ~20-25kb sized files in dropbox. Syncing it with my local machine is a pain in the ass (API limits syncing to 4-5 files per second, so I will need at least 96 hours just to take out everything from dropbox).

Recently I started looking into self hosted cloud solutions. I fiddled around with nextcloud but it's client limits syncing to 5-6 files per second as well. Accessing nextcloud via webdav from a rclone client I was able to up the syncing speed to 15-25 files per second but that's all.

Any solutions where I should host 2mil of files and would be able to sync them fast to my own machine with some cloud client when I need it to?

question

sync

cloud
Favorite
Ranter
Comments
  • 0
    5y
    What do you need dropbox for? I might have a solution, but it depends on your use-case
  • 0
    5y
    @netikras What do you mean? I don't need dropbox anymore. I piled files in it and now it's became sluggish. What is your potential solution? Another cloud?
  • 1
    5y
    @zemaitis You're complaining about slow performance because there are many files. So why not reduce amount of files to sync?

    Make containers, like truecrypt filesystems, and store everything in them. You will have less files to sync.

    Amount of data per-file (and per-change) will be larger, but if you can sync on-demand it does not really matter.

    This will work perfectly if you are using cloud storage as an archive or a way to transfer data between devices, as access to files is no longer flexible. THAT is why asked you what's your use case.

    Did that with a part of my dropbox storage.
  • 1
    5y
    Well if you are going to self host then rsync over SSH is a way that works nice. Or maybe compressing the files with tar then untar on end using a bash script?
  • 0
    5y
    +1 for rsync incremental backups
Related Rants
devRant © 2021 Hexical Labs LLC
Privacy Policy  |  Terms of Service
Add Comment