So I've got this old external hard drive that's on the verge of retirement. I bought a new one to move the content of the old one to, but it's a lot of data. (1+ TB)

Does anyone know of a good program that can do that reliably?

@irene I don't know that I know the difference actually. :/ All I know is I'd like all the data transferred as losslessly as humanly possible. There are a lot of photos and such from years back on that thing.

I just use Rsync of a target folder with the old drive in a HDD dock.

Unless you want to copy the OS and everything on the drive aswell?

Which OS are you using?

@C0D4 No, this is just an external drive with photos and videos and all kinds of crap on it, no OS installed, just file storage.

@Navigatr oh ok.

If it's a significant amount of files I just boot up a Linux VM, mount the drives and use Rsync to send my files across.

If you want to watch it just shove the --progress flag on.

@C0D4 I do have a laptop running Linux Mint but it's not very powerful. Do you think that would affect the process much at all if I did the transfer on that?

I mean, I assume more cores = faster transfer, but yeah. It'd be less work ahead of time than setting up a VM.

@Navigatr it would probably be slow.

But then, probably no worse then windows so 🤷‍♂️

I'd just give it all the resources you can, especially ram, and see how it goes.

@C0D4 I'll certainly try it out at the very least. :) Thanks for all the help!

@irene as long as it's only reading from the dying disk it's usually fine.

@C0D4 @irene I think the disk itself is in an alright condition, but the cable connections can be a bit iffy. Once it's got a connection and as long as you don't touch it it's usually fine. I'd just like to get the stuff off of it before it breaks down so I don't have to pay for one of those data retrieval services. :P

This question has been posted an hour ago. During that hour you would have copied all the files with a simple copy-paste...

@netikras As far as I know that would require me to have a hell of a lot of RAM.

Edit: And it *definitely* wouldn't take me just an hour to copy-paste over a TB of data.

@Navigatr how is RAM relevant here? You are not copying into RAM... You are copying into another device.

1TB is not that much you know...

@netikras Like I mentioned above that heavily depends on how good your CPU is, and I wouldn't call mine high-end, really. :P

@Navigatr so is it RAM or CPU now? :D

I thought these newer HDDs use DMA to exchange data which makes your CPU slowness kind of irrelevant

and I'm pretty sure you would have finished copying within that hour if you were cloning your disk (`dd`).

@netikras i assumed @Navigatr was transferring between 2 external drives, that will copy to ram and then to second disk.

@netikras See, I haven't studied these things. XD I've picked up a lot of stuff as I go and as far as I know, if you're copying large amounts of data and want to ensure nothing is lost, it's good to have a dedicated program do the work. And maybe not doing it on Windows.

@C0D4 That's exactly what I'm doing. :) That's why I assumed as much.

@C0D4 ?? wut??

Do you mean that Windows copies all the content to RAM at once before beginning to write out? :)

IDK, maybe Windows is spoiled THAT much... But Linux does have buffers for that purpose.

@netikras no, windows caches everything as it reads data.

So although it's a single transfer, Windows goes "oh I'm a bright cookie, let me shove that in ram on the off chance your going to look at that again" and then as your ram fills it starts removing the older content.

I haven't paid any attention on Linux for these kinds of transfers, I just set Rsync up, walk away and let it run.

@C0D4 yupp, Linux does that too :) It's caching read data as well. It's a philosophy I have seen in many UNIX machines: unused ram is wasted ram.

that's why `free -h` output has that 'buff/cache' column. It's mostly reclaimable memory and has positive (best case) to next-to-none-adverse (worst case) impact on performance.

I don't think caching and low RAM should be a problem here...

FFS, I used to copy 30GB+ picture files from one USB device to another with my Intel Atom netbook with 1GB of RAM..

@netikras I get it caching if I'm actually using that data, but if it's just being transferred.. meh, lousy philosophy.

@Navigatr you started that transfer yet?

@C0D4 Well the machine does not know whether you are going to use it or not :) It also does not know whether it will have to retry some action requiring this data (like writing to HDD1) or not...

@netikras I don't know, the whole coping in bulk would be a good basis to work that out from.

It's not like your going to open 30GB of files moments after it's been copied.

@C0D4 I'm not in a position to do that at this very moment, but I'll get to it later when I have access to my computer. :)

I can recommend TeraCopy by Code Sector if you're using Windows. It does checksum (e.g. CRC) verification of each file as it copies. Free version available but I liked it enough to purchase the Pro version (no affiliation).

For Linux/Unix users I'd recommend either partitioning, formatting, then rsync, or Clonezilla. If you are happy with the factory-provided partitioning/formatting, you may skip those steps.

Under Windows, copying everything using Windows Explorer should work just fine.

Rsync is the correct tool if you have a broken cable. On the second try it can efficiently copy only the data that was not already sent.