Files corrupted after converting HFS+ volume to encrypted APFS

Question

Recently I attempted to convert a (non-boot) hard drive I use for backups from HFS+ to APFS (encrypted). I started by converting the drive to APFS (not encrypted):

diskutil apfs convert /Volumes/Archive

I don't remember clearly, but I think I used the Disk Utility app for that step, not the command line. This appeared to go through quite quickly and successfully. All good. Next I ran a command to encrypt the volume:

diskutil apfs encryptVolume disk7s1 -user disk 

I entered a new encryption key when prompted and proceeded to start the encryption process. I definitely did this step on the command line. I don't have the terminal output anymore, but it didn't return any errors and appeared to be working as expected.

In addition to various backup copies of files stored elsewhere, the drive contains an HFS+ formatted disk image (Backup.sparsebundle) that is used as a Time Machine backup destination. I noticed that this disk image had become unmounted in the process of converting and encrypting the disk, so I tried to remount it. To my surprise and alarm, this caused macOS to crash hard and reboot.

After rebooting I checked in on the disk encryption process, which appeared to be still in progress:

Container disk7 E0...
====================================================
APFS Container Reference:     disk7
Size (Capacity Ceiling):      4000443056128 B (4.0 TB)
Minimum Size:                 2472421744640 B (2.5 TB)
Capacity In Use By Volumes:   2396837355520 B (2.4 TB) (59.9% used)
Capacity Not Allocated:       1603605700608 B (1.6 TB) (40.1% free)
|
+-< Physical Store disk6s2 D2...
|   -----------------------------------------------------------
|   APFS Physical Store Disk:   disk6s2
|   Size:                       4000443056128 B (4.0 TB)
|
+-> Volume disk7s1 B7...
    ---------------------------------------------------
    APFS Volume Disk (Role):   disk7s1 (No specific role)
    Name:                      Archive (Case-insensitive)
    Mount Point:               /Volumes/Archive
    Capacity Consumed:         2396365815808 B (2.4 TB)
    Encryption Progress:       10.0% (Unlocked)

After observing for a while I thought the encryption had stalled, but after hours it progressed to 11.0%, so I assume it is encrypting the drive in the background...very slowly. A week later it's only at 32.0%. Very slow indeed.

Unfortunately, while the directory structure is intact, the files on the drive are unreadable. Maybe this is why the disk image did not mount and caused a hard crash? For example, there is a text file that contains logs from an rsync job that appears to be corrupted. Opening it with vim it shows no content and an error message in the status bar:

"/Volumes/Archive/rsync-stats.txt" [readonly][READ ERRORS] 0L, 0C

I am able to view the (uncorrupted) content of some files with iBoysoft Data Recovery.app, although contrary to what the website says, the program will not recover files for free. Some of the files that I was able to preview with iBoysoft a week ago now appear to be corrupted, even when previewed with iBoysoft—maybe this is due to the encryption process? None of them have ever been viewable outside of iBoysoft since the drive was encrypted. Was this corruption caused by a bug in APFS encryption?

Verifying the drive with diskutil shows no errors (ran this a few minutes ago):

$ sudo diskutil verifyVolume disk7s1
Password:
Started file system verification on disk7s1 Archive
Verifying file system
Volume was successfully unmounted
Performing fsck_apfs -n -x /dev/rdisk7s1
Checking volume
Checking the container superblock
Checking the EFI jumpstart record
Checking the space manager
Checking the object map
Checking the APFS volume superblock
Checking the object map
Checking the fsroot tree
Checking the snapshot metadata tree
Checking the extent ref tree
Checking the snapshots
Verifying allocated space
The volume /dev/rdisk7s1 appears to be OK
File system check exit code is 0
Restoring the original state found as mounted
Finished file system verification on disk7s1 Archive

Unfortunately I do not have a backup of my backup drive. And yes, it was silly to do this without making a backup first. It's not the end of the world if I lose the files, but I'd rather not lose them if I don't have to.

The machine is running macOS 10.13.6

Two questions:

1. Can I recover the data on this drive, preferably with tools from Apple or the open source community? How?
2. Is there a way to tell macOS to prioritize the encryption process to speed things up? 32% after a week is pretty sad, and most of the time the computer is sitting idle, so it has lots of resources to spare.

Edit:

3. Are there tools to troubleshoot what went wrong with the encryption process that caused files on the drive to become unreadable?

Answer 1

There are multiple challenges with the process you have gone through.

First of the entire premise of converting the drive seems to be a bad idea. You're using Time Machine for backups and want to convert the drive to APFS. But Time Machine doesn't support APFS at all - so even if you get the drive converted fully and correctly, then it won't work with Time Machine afterwards.

Regarding backup process: This process is indeed slow (or at least time consuming). There's no real way of telling macOS to speed things up, but you can ensure that the computer hasn't got much else to do. I have also seen that simply moving the mouse pointer around a little has a positive effect on the speed. Note that you can use the drive as usual while it is encrypting, and there's nothing that will be "fixed" by the drive finishing encryption.

You ask if the corruption occured due to a bug in APFS encryption. While it is possible, it doesn't sound very likely no. Most likely you had a pre-existing problem.

Whether or not you can recover data from the drive is impossible to say from your question. You can use various recovery tools and/or third party recovery services, they might be able to recover some of your data - but there are no guarantees.

All in all I think your experiences indicate that you have a hardware problem with your drive. I would suggest immediately shutting down your computer (you should really have done this a week ago when the problems started, but that's too late now). After shutting it down, use a different computer to take a bit-for-bit copy of the drive contents. Then run your data recovery programs, etc. on the copy instead of the original drive.

If you end up wiping the drive and starting over, I would suggest very thoroughly testing the drive before assuming it works. Again, it could very likely be a hardware problem with the drive.

Answer 2

The answer is: wait for it. No really, that's the answer, just wait for the encryption to get to 100%. It took a very long time (over a month! in my case), but now that the encryption process is complete I'm happy to report that all of the files on the drive are readable again. I have no explanation for why the files were unreadable while encryption was in progress. I don't think it's supposed to work that way.