So I finally recovered most of my data but is was painful long experience. Here is what helped in the end:
I don't know for sure exactly, but after another unsuccessful try to run first aid on my hard disk I tried to unmount/mount it manually using the terminal in recovery mode (booted from the external USB):
$ diskutil unmount /dev/disk2 # this was successful
$ diskutil mount /dev/disk2. # this failed
However after rebooting again into recovery mode suddenly first aid run through successfully. So don't give up too early. Sometimes rebooting seems to fix something. Maybe also my reseting of the nvram via terminal nvram -c helped. Maybe even using disk util in English instead of my mother language helped. I also realized one should always click on View -> Show All Devices in Disk Utility and also run first aid on the container. I tried to reboot and see if my Mac is booting again but it still hang. Since at that time it was very clear to me that I had some hardware issue with my drive and because I couldn't open the iMac I switched it off to cool down. I also turned it on its head and moved it a little bit in a desperate move in case it was a slightly loose cable or something like that. After that I rebooted into recovery mode (this time from internal hard disk).
Then I tried copying my data using
cp -npRv /Volumes/Macintosh HD/<important_dir> /Volumes/BACKUP_USB_APFS
and it worked. To make sure my iMac would not sleep and the copy process would not stop I started in a separate terminal window
$ caffeinate
And to be absolutely sure I also configured the global power settings and switched of sleep and hard disk sleep
$ pmset -a sleep 0
$ pmset -a disksleep 0
However the copy process was hanging/slow at several file (presumably because of bad blocks or some hard disk damage). But I realized I could speed up the process by starting the same copy process over and over in separate terminal windows. The option -n makes sure, that no file will be overwritten, that means no copy process will be done twice.
After several hours (I ran it through the night) I finally had copied most of the data successfully to my external USB hard drive. Now to be very sure my multiple copy processes didn't mess up things I decided to run rsync in addition. rsync is coming with Mac OS X, but it is not in your path when working in recovery mode. So you have first switch to the /usr/bin directory before being able to use it. At first I did a dry run to see which file would be modified:
$ cd /Volumes/Macintosh HD/usr/bin
$ ./rsync -ai --exclude="._*" --dry-run /Volumes/Macintosh HD/<important_dir> /Volumes/BACKUP_USB_APFS | grep "^>" >> /Volumes/BACKUP_USB_APFS/lost_data.txt
This will show all files which will be added by rsync (-i/--itemize option will show a > for each file rsync plans to modify, which we filter with grep). Here I discovered an interesting quirk of the version of rsync coming with Mac OS. When one is using the option -E rsync will not allow to exclude dot files, meaning --exclude="._*" doesn't work. But for copying I used the -E option to make sure to get the extended attributes, which cp ignored.
$ rsync -vaEP /Volumes/Macintosh HD/<important_dir> /Volumes/BACKUP_USB_APFS
It should be noted that rsync is running much much slower than cp on a damaged hard disk. Especially when trying to copy huge files (movies) rsync takes a lot of time until it finally gives up. The whole process was running for 1,5 days. So I always recommend to first use cp and only afterwards use rsync.
So that is it. I managed to recover most of the data with that from my broken hard drive. But I do want to also mention a lot of things that didn't work:
- Apple Transfer mode (press
T at boot time) didn't work. The destination laptop showed the drive of my broken iMac but then crashed quickly when trying to access it.
- Apple Diagnostics (press
D at boot time) didn't work. It only showed error -2002D. This was weird, because my Wifi worked even from recovery mode.
- I tried to use Rclone, because in opposition to
rsync and cp it is multi-threaded. You can install it in recovery mode by downloading it with curl curl -O https://downloads.rclone.org/v1.59.1/rclone-v1.59.1-osx-amd64.zip, unzipping the file with \usr\bin\unzip and just start the executable. However it turned out to be even slower than rsync in my case.
- I tried using Disk Drill in recovery mode as described here. However the copy process simply stopped, because the hard drive "vanished" during the process. Somehow it got unmounted. That happened several times so I gave up on it. Also DiskDrill by default doesn't copy hidden files (you have to find the menu point to activate this first) and seems to not copy all meta data correctly.
- Creating a bootable USB drive failed at first. I tried downloading the Catalina installer (on another laptop) from App Store as described by Apple. But the download failed with an error. So I tried to download it via commandline, but it also failed with a weird error. The only thing that worked in the end was using mist-cli. Here is how to create a bootable USB drive on command line quickly with mist-cli
Do this:
$ brew install mist
$ mist list installer
$ sudo mist download installer 10.15.7 application
$ sudo /Users/Shared/Mist/Install\ macOS\ Catalina.app\ 10.15-19H15.app/Contents/Resources/createinstallmedia --volume /Volumes/<MyUSBDrive>
