Slow transfer between two external drives

Question

Copying data to a USB external drive is much slower than expected.

I have a 6TB external RAID drive that holds my photos, and I'm trying to do an archive copy of this data to another external USB drive.

Both are HDD drives, formatted with HFS+. I have turned off the Spotlight indexing for both drives. I'm using a MacBookPro with fast USB-C ports, and USB 3.0 or better ports on both external drives. All cables are 3.0 (SS) cables or better.

The file transfer started at about 200MB/s then slowed to around 20MB/s and sometimes even less. I'm using the Activity Monitor - Disk to display the read/write speed.

I think the initial transfer speed was the write buffers on the target drive filling up. I'm expecting a real-life transfer speed for USB 3.0 to be around 100MB/s.

Any idea why this is running so slowly. How can I figure out the bottleneck?

Jim

Answer 1

Assuming there is no fault with any hardware or software, the following are some of the major factors that can contribute to a copy speed proceeding slower than the maximum transfer speed that a hard drive is capable of:

1. Copying large numbers of small files with a single-threaded copying process... The overhead of simply opening each file for reading and creating each file for writing plus updating the HFS+ file system metadata takes extra time, which can slow single-threaded copy processes like drag-and-drop in the Finder considerably from this extra file system overhead. Some utilities such as the command-line tool rsync can speed up this process by pre-fetching files into its own memory buffer then writing the files to the destination drive from this buffer, thereby reducing the effect somewhat. Some third-party tools such as SuperDuper!, Beyond Compare, and others also perform some of this buffering to try and speed the file copy process but regardless of the utility used, copying large numbers of small files will naturally result in a lower overall throughput than if large files were being copied, due to this extra file system overhead.
2. Copying large files from a hard drive with a severely fragmented file system... Copying at the file-level will open the file on the source drive then copy that file from beginning-to-end to the destination drive, but the file may not be stored contiguously on the source hard drive. In that case, the hard drive read-write head may have to seek to many different areas of the disk in order to read the file in-order, slowing file transfer considerably. This is more common for files that have changed a lot on disk over the years, and not so much for archival files that were created on the drive once then left (like photos) so this is probably not the case here.
3. Copying files from a newer shingled-magnetic-recording (SMR) hard drive... The newest SMR hard drives cannot directly read the area on the drive that the HFS+ file system says contains the file. It must first read an entire track all the way around the disk, then pull out the portion of the data from that track that contains the file data that was requested. This is especially detrimental to copy speed as it mimics the above "fragmented hard drive" example, except it ensures that nearly every file behaves as if it is "fragmented" so unless you are copying the files off of the drive in the exact same order that they were all created, you will encounter severe slow down in copy speed. This is especially common among the largest low-cost external USB drives available today.

Answer 2

Can use iostat to debug / watch the performance over time - e.g. (summarise over 30 second intervals)

# iostat 30
              disk0               disk2       cpu    load average
    KB/t  tps  MB/s     KB/t  tps  MB/s  us sy id   1m   5m   15m
   24.73    8  0.20    30.90    0  0.00  10  7 82  1.40 1.67 2.02
    5.54  935  5.06     0.00    0  0.00  13 20 67  1.48 1.68 2.02
   10.44  594  6.06     0.00    0  0.00  16 22 62  1.49 1.67 2.01

You just need to check which is which .. e.g. (I have an internal SSD and an external spindle)

# diskutil list
/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *251.0 GB   disk0
   1:                        EFI EFI                     314.6 MB   disk0s1
   2:                 Apple_APFS Container disk1         250.7 GB   disk0s2

/dev/disk1 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +250.7 GB   disk1
                                 Physical Store disk0s2
   1:                APFS Volume Macintosh HD — Data     112.4 GB   disk1s1
   2:                APFS Volume Preboot                 91.6 MB    disk1s2
   3:                APFS Volume Recovery                529.0 MB   disk1s3
   4:                APFS Volume VM                      4.3 GB     disk1s4
   5:                APFS Volume Macintosh HD            11.3 GB    disk1s5

/dev/disk2 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *3.0 TB     disk2
   1:                        EFI EFI                     209.7 MB   disk2s1
   2:                  Apple_HFS BACKUP                  3.0 TB     disk2s2