10

I need to copy 20TB of data onto a thunderbolt array. The box where the data exists does not have a thunderbolt connection, so I will need to utilize the local 1GB network for this. (Yes, it will take forever).

I tried to use Filezilla/sftp, but it crashed when the queue grew super large. Is rsync or scp the way to go?

  • 2
    Can you get the drive out of the old device and attach it directly? – user93464 Oct 4 '14 at 1:14
  • Concur with the above comment. Physical transfer has the largest bandwidth. – Siyuan Ren Oct 7 '14 at 14:28
  • I love Carbon Copy Cloner, which is a nice GUI utility based on rsync. The developer forked rsync and made lots of improvements. I can't speak to the speed vs rsync and/or tar or whatever, but it's my go-to for any kind of data transfer, where I want to know without a doubt that my data's good once all is said and done. And if it's not, CCC will tell me. – Harv Oct 7 '14 at 18:45
  • I'm surprised there are no references to Fedex. – tedder42 Oct 7 '14 at 21:17
16

rsync is a good way to go (scp is pretty much the same with fewer features). You may want to use the -Z option, which will enable zlib compression. Depending on how fast your drives/computer are, it may be faster than sending uncompressed, i.e. if your network link is saturated. You may also want the archive mode option, -a which will preserve symlinks, permissions, and creation/modification times, as well as copy directories recursively. Depending on what you're copying you might want -E which preserves extended attributes and mac resource forks. Finally, --progress will show you progress information.

  • 11
    rsync is also good at restarting copying if there is an interruption. – Lee Joramo Oct 3 '14 at 18:18
  • 3
    Seconding @LeeJoramo, being able to pick up where a failed transfer left off is extremely important. A transfer this large will fail at some point, and you want to make sure that you don't lose whatever progress you've made to that point. – blahdiblah Oct 3 '14 at 18:41
6

While not as ubiquitous as rsync, I have in the past used a tool call "mpscp" - http://www.sandia.gov/MPSCP/mpscp_design.htm

From Sandia National Labs, it's a file copy tool that runs over SSH that is specially optimized to saturate high-speed networks between close systems (such as copying terabytes of data between two supercomputers at the same site, connected via 10Gb+ or Infiniband). It works well, but can be a bit of a pain to setup. In testing, I've easily seen it run 2x-3x faster than rsync.

3

Use rsync and consider using it with rsyncd. If you use rsync without rsyncd, you're stuck using ssh, which means using some kind of encryption. You're probably copying the data from an older machine to a newer machine and the older machine may not have the CPU grunt to encrypt the data for transmission fast enough to keep a gigabit Ethernet link saturated. Test transferring batches of files using both methods and see which way is faster.

For the same reason I would advise testing use of rsync's compression option before committing to using it. Compression is another CPU intensive activity that might not be able to keep up with gigabit Ethernet speeds when attempted on older hardware. rsync is a fifteen year old program, written back when the majority of people even in first world countries accessed the Internet via dialup modem. Network bandwidth vs. CPU tradeoffs were much different then.

  • You're right that rsync defaults to using ssh, but that's not the only option. You can make it use rsh instead using the -e option. With the same option, you can change ssh's options to use less cpu-intensive encryption: -e 'ssh -c arcfour,blowfish-cbc' or the like. Whether this makes any speed difference with a modern machine, I don't know, but a quick benchmark may be worthwhile, especially with 20 TB of files. – Michael D. M. Dryden Oct 4 '14 at 1:01
3

Is this 20Tb packaged in a small number of large files (like video, monster database) or millions of smaller files?

If lots of small files I would go with rsync for restartability or a piped tar stream for efficiency (one network connection for the lot, start again from the beginning if it fails)

tar -cf - * | ( cd newhome; tar -xf - )

remote folder must be mounted.

Could you directly attach the new array with a different interface? Local rsync doesn't use ssh so you remove that failure point. Yes, Firewire 800 is slower than gigabit ethernet but you cannot load ethernet to 100% - it might be faster by firewire. FYI you can also network firewire if the boxes are close enough. Add the interface in system preferences -> network.

0

Another option would be to try Bittorrent Sync (http://www.bittorrent.com/sync). I've used it to sync family photos and videos between members of our family across the WAN but there's no reason it won't work for local network. It uses peer-to-peer connections so the data would not be going through a server like it would if you tried to use something like dropbox (not that I think you have 20TB of dropbox space or want to wait that long to upload that much data!)

It's also supported on multiple platforms so has more flexibility than rsync and tar.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .