I need to copy 20TB of data onto a thunderbolt array. The box where the data exists does not have a thunderbolt connection, so I will need to utilize the local 1GB network for this. (Yes, it will take forever).
I tried to use Filezilla/sftp, but it crashed when the queue grew super large. Is rsync or scp the way to go?
rsync is a good way to go (scp is pretty much the same with fewer features). You may want to use the -Z option, which will enable zlib compression. Depending on how fast your drives/computer are, it may be faster than sending uncompressed, i.e. if your network link is saturated. You may also want the archive mode option, -a which will preserve symlinks, permissions, and creation/modification times, as well as copy directories recursively. Depending on what you're copying you might want -E which preserves extended attributes and mac resource forks. Finally, --progress will show you progress information.
While not as ubiquitous as rsync, I have in the past used a tool call "mpscp" - http://www.sandia.gov/MPSCP/mpscp_design.htm
From Sandia National Labs, it's a file copy tool that runs over SSH that is specially optimized to saturate high-speed networks between close systems (such as copying terabytes of data between two supercomputers at the same site, connected via 10Gb+ or Infiniband). It works well, but can be a bit of a pain to setup. In testing, I've easily seen it run 2x-3x faster than rsync.
Use rsync and consider using it with rsyncd. If you use rsync without rsyncd, you're stuck using ssh, which means using some kind of encryption. You're probably copying the data from an older machine to a newer machine and the older machine may not have the CPU grunt to encrypt the data for transmission fast enough to keep a gigabit Ethernet link saturated. Test transferring batches of files using both methods and see which way is faster.
For the same reason I would advise testing use of rsync's compression option before committing to using it. Compression is another CPU intensive activity that might not be able to keep up with gigabit Ethernet speeds when attempted on older hardware. rsync is a fifteen year old program, written back when the majority of people even in first world countries accessed the Internet via dialup modem. Network bandwidth vs. CPU tradeoffs were much different then.
-e option. With the same option, you can change ssh's options to use less cpu-intensive encryption: -e 'ssh -c arcfour,blowfish-cbc' or the like. Whether this makes any speed difference with a modern machine, I don't know, but a quick benchmark may be worthwhile, especially with 20 TB of files.
Commented
Oct 4, 2014 at 1:01
Is this 20Tb packaged in a small number of large files (like video, monster database) or millions of smaller files?
If lots of small files I would go with rsync for restartability or a piped tar stream for efficiency (one network connection for the lot, start again from the beginning if it fails)
tar -cf - * | ( cd newhome; tar -xf - )
remote folder must be mounted.
Could you directly attach the new array with a different interface? Local rsync doesn't use ssh so you remove that failure point. Yes, Firewire 800 is slower than gigabit ethernet but you cannot load ethernet to 100% - it might be faster by firewire. FYI you can also network firewire if the boxes are close enough. Add the interface in system preferences -> network.
Another option would be to try Bittorrent Sync (http://www.bittorrent.com/sync). I've used it to sync family photos and videos between members of our family across the WAN but there's no reason it won't work for local network. It uses peer-to-peer connections so the data would not be going through a server like it would if you tried to use something like dropbox (not that I think you have 20TB of dropbox space or want to wait that long to upload that much data!)
It's also supported on multiple platforms so has more flexibility than rsync and tar.
