Skip to main content
edited body
Source Link
Jivan Pal
  • 1.3k
  • 12
  • 21

There is a particular fsck error:

** Checking the object map.
error: (oid 0xd31c1) om: btn: found zeroed-out block
   Object map is invalid.

Here, om refers to the object map of the macOS volume, and btn refers to a B-tree node in that object map. Evidently, part of the node has been zeroed-out, leading to some or all of the dentries for /Users/jivan being inaccessible.

I developed some tools to inspect the APFS container, in the hopes that older versions of the object map and other file-system structures were intact (as referenced by an older APFS transaction), thereby allowing me to access my files. Using these tools, I indeed found that a few nodes in the file-system root B-tree for my main APFS volume had been zeroed out. Thanks to APFS's copy-on-write/transaction-based behaviour, I was able to search the entire disk for older versions of these missing nodes, and successfully found recent instances of them — except for the particular leaf node that contains the file-system records for /Users/jivan, so its contents cannot be directly determined. Just my luck(!) However, I was able to see that /Users/jivan had an ID of 0xb54a8, and thus search for nodes which contained dentries for items whose parent ID was also 0xb54a8; these nodes were then the ones which listed the contents of /Users/jivan.

In order to more easily do an automated recovery, I reconstructed the missing internal node of the file-system B-tree, and then used my apfs-recover tool to actually get each file. For example, to recover /Users/jivan/Documents/my file.pdf, I can do:

apfs-recover /dev/disk2s2 0 "/Users/jivan/Documents/my file.pdf" > "~/Desktop/my file.pdf"

Rather than run such a command for each file, I wrote a Bash script, pull.sh, which, when given a target recovery directory and a file which lists paths to files to attempt to recover, runs apfs-recover for each such filepath and outputs the result to a corresponding path in the recovery directory. For example, if the contents of filepaths.txt are

/Users/jivan/Documents/my doc.pdf
/Users/jivan/Pictures/my pic.jpg

then running pull.sh ~/DekstopDesktop/RECOVERY filepaths.txt recovers the files to the following paths:

~/Desktop/RECOVERY/Users/jivan/Documents/my doc.pdf
~/Desktop/RECOVERY/Users/jivan/Pictures/my pic.jpg

I added the desired entries in filepaths.txt with some programmatic assistance, and was then able to successfully recover the vast majority of my files. For any particularly important files which this script fails to recover (due to bugs in the software I've written or additional malformed/missing APFS structures on the affected disk) I'll have to dig deeper, but this is effectively solved now.

All tools mentioned are available in the Git repo.

There is a particular fsck error:

** Checking the object map.
error: (oid 0xd31c1) om: btn: found zeroed-out block
   Object map is invalid.

Here, om refers to the object map of the macOS volume, and btn refers to a B-tree node in that object map. Evidently, part of the node has been zeroed-out, leading to some or all of the dentries for /Users/jivan being inaccessible.

I developed some tools to inspect the APFS container, in the hopes that older versions of the object map and other file-system structures were intact (as referenced by an older APFS transaction), thereby allowing me to access my files. Using these tools, I indeed found that a few nodes in the file-system root B-tree for my main APFS volume had been zeroed out. Thanks to APFS's copy-on-write/transaction-based behaviour, I was able to search the entire disk for older versions of these missing nodes, and successfully found recent instances of them — except for the particular leaf node that contains the file-system records for /Users/jivan, so its contents cannot be directly determined. Just my luck(!) However, I was able to see that /Users/jivan had an ID of 0xb54a8, and thus search for nodes which contained dentries for items whose parent ID was also 0xb54a8; these nodes were then the ones which listed the contents of /Users/jivan.

In order to more easily do an automated recovery, I reconstructed the missing internal node of the file-system B-tree, and then used my apfs-recover tool to actually get each file. For example, to recover /Users/jivan/Documents/my file.pdf, I can do:

apfs-recover /dev/disk2s2 0 "/Users/jivan/Documents/my file.pdf" > "~/Desktop/my file.pdf"

Rather than run such a command for each file, I wrote a Bash script, pull.sh, which, when given a target recovery directory and a file which lists paths to files to attempt to recover, runs apfs-recover for each such filepath and outputs the result to a corresponding path in the recovery directory. For example, if the contents of filepaths.txt are

/Users/jivan/Documents/my doc.pdf
/Users/jivan/Pictures/my pic.jpg

then running pull.sh ~/Dekstop/RECOVERY filepaths.txt recovers the files to the following paths:

~/Desktop/RECOVERY/Users/jivan/Documents/my doc.pdf
~/Desktop/RECOVERY/Users/jivan/Pictures/my pic.jpg

I added the desired entries in filepaths.txt with some programmatic assistance, and was then able to successfully recover the vast majority of my files. For any particularly important files which this script fails to recover (due to bugs in the software I've written or additional malformed/missing APFS structures on the affected disk) I'll have to dig deeper, but this is effectively solved now.

All tools mentioned are available in the Git repo.

There is a particular fsck error:

** Checking the object map.
error: (oid 0xd31c1) om: btn: found zeroed-out block
   Object map is invalid.

Here, om refers to the object map of the macOS volume, and btn refers to a B-tree node in that object map. Evidently, part of the node has been zeroed-out, leading to some or all of the dentries for /Users/jivan being inaccessible.

I developed some tools to inspect the APFS container, in the hopes that older versions of the object map and other file-system structures were intact (as referenced by an older APFS transaction), thereby allowing me to access my files. Using these tools, I indeed found that a few nodes in the file-system root B-tree for my main APFS volume had been zeroed out. Thanks to APFS's copy-on-write/transaction-based behaviour, I was able to search the entire disk for older versions of these missing nodes, and successfully found recent instances of them — except for the particular leaf node that contains the file-system records for /Users/jivan, so its contents cannot be directly determined. Just my luck(!) However, I was able to see that /Users/jivan had an ID of 0xb54a8, and thus search for nodes which contained dentries for items whose parent ID was also 0xb54a8; these nodes were then the ones which listed the contents of /Users/jivan.

In order to more easily do an automated recovery, I reconstructed the missing internal node of the file-system B-tree, and then used my apfs-recover tool to actually get each file. For example, to recover /Users/jivan/Documents/my file.pdf, I can do:

apfs-recover /dev/disk2s2 0 "/Users/jivan/Documents/my file.pdf" > "~/Desktop/my file.pdf"

Rather than run such a command for each file, I wrote a Bash script, pull.sh, which, when given a target recovery directory and a file which lists paths to files to attempt to recover, runs apfs-recover for each such filepath and outputs the result to a corresponding path in the recovery directory. For example, if the contents of filepaths.txt are

/Users/jivan/Documents/my doc.pdf
/Users/jivan/Pictures/my pic.jpg

then running pull.sh ~/Desktop/RECOVERY filepaths.txt recovers the files to the following paths:

~/Desktop/RECOVERY/Users/jivan/Documents/my doc.pdf
~/Desktop/RECOVERY/Users/jivan/Pictures/my pic.jpg

I added the desired entries in filepaths.txt with some programmatic assistance, and was then able to successfully recover the vast majority of my files. For any particularly important files which this script fails to recover (due to bugs in the software I've written or additional malformed/missing APFS structures on the affected disk) I'll have to dig deeper, but this is effectively solved now.

All tools mentioned are available in the Git repo.

Compile answer from prior question updates
Source Link
Jivan Pal
  • 1.3k
  • 12
  • 21

Answering this question myself soThere is a particular fsck error:

** Checking the object map.
error: (oid 0xd31c1) om: btn: found zeroed-out block
   Object map is invalid.

Here, om refers to the object map of the macOS volume, and btn refers to a B-tree node in that object map. Evidently, part of the node has been zeroed-out, leading to some or all of the dentries for /Users/jivan being inaccessible.

I can mark it as solved;developed some tools to inspect the APFS container, in the hopes that older versions of the object map and other file-system structures were intact (as referenced by an older APFS transaction), thereby allowing me to access my files. Using these tools, I indeed found that a few nodes in the file-system root B-tree for my main APFS volume had been zeroed out. Thanks to APFS's copy-on-write/transaction-based behaviour, I was able to search the entire disk for older versions of these missing nodes, and successfully found recent instances of them — except for the particular leaf node that contains the file-system records for /Users/jivan, so its contents cannot be directly determined. Just my luck(!) However, I was able to see that /Users/jivan had an ID of 0xb54a8, and thus search for nodes which contained dentries for items whose parent ID was also 0xb54a8; these nodes were then the updateones which listed the contents of /Users/jivan.

In order to more easily do an automated recovery, I reconstructed the question made on 2019missing internal node of the file-11system B-14tree, and then used my apfs-recover tool to actually get each file. For example, to recover /Users/jivan/Documents/my file.pdf, I can do:

apfs-recover /dev/disk2s2 0 "/Users/jivan/Documents/my file.pdf" > "~/Desktop/my file.pdf"

Rather than run such a command for each file, I wrote a Bash script, pull.sh, which, when given a target recovery directory and a file which lists paths to files to attempt to recover, runs apfs-recover for each such filepath and outputs the result to a corresponding path in the recovery directory. For example, if the contents of filepaths.txt are

/Users/jivan/Documents/my doc.pdf
/Users/jivan/Pictures/my pic.jpg

then running pull.sh ~/Dekstop/RECOVERY filepaths.txt recovers the files to the following paths:

~/Desktop/RECOVERY/Users/jivan/Documents/my doc.pdf
~/Desktop/RECOVERY/Users/jivan/Pictures/my pic.jpg

I added the desired entries in filepaths.txt with some programmatic assistance, and was then able to successfully recover the vast majority of my files. For any particularly important files which this script fails to recover (due to bugs in the software I've written or additional malformed/missing APFS structures on the affected disk) I'll have to dig deeper, but this is effectively solved now.

All tools mentioned are available in the Git repo.

Answering this question myself so that I can mark it as solved; see the update to the question made on 2019-11-14.

There is a particular fsck error:

** Checking the object map.
error: (oid 0xd31c1) om: btn: found zeroed-out block
   Object map is invalid.

Here, om refers to the object map of the macOS volume, and btn refers to a B-tree node in that object map. Evidently, part of the node has been zeroed-out, leading to some or all of the dentries for /Users/jivan being inaccessible.

I developed some tools to inspect the APFS container, in the hopes that older versions of the object map and other file-system structures were intact (as referenced by an older APFS transaction), thereby allowing me to access my files. Using these tools, I indeed found that a few nodes in the file-system root B-tree for my main APFS volume had been zeroed out. Thanks to APFS's copy-on-write/transaction-based behaviour, I was able to search the entire disk for older versions of these missing nodes, and successfully found recent instances of them — except for the particular leaf node that contains the file-system records for /Users/jivan, so its contents cannot be directly determined. Just my luck(!) However, I was able to see that /Users/jivan had an ID of 0xb54a8, and thus search for nodes which contained dentries for items whose parent ID was also 0xb54a8; these nodes were then the ones which listed the contents of /Users/jivan.

In order to more easily do an automated recovery, I reconstructed the missing internal node of the file-system B-tree, and then used my apfs-recover tool to actually get each file. For example, to recover /Users/jivan/Documents/my file.pdf, I can do:

apfs-recover /dev/disk2s2 0 "/Users/jivan/Documents/my file.pdf" > "~/Desktop/my file.pdf"

Rather than run such a command for each file, I wrote a Bash script, pull.sh, which, when given a target recovery directory and a file which lists paths to files to attempt to recover, runs apfs-recover for each such filepath and outputs the result to a corresponding path in the recovery directory. For example, if the contents of filepaths.txt are

/Users/jivan/Documents/my doc.pdf
/Users/jivan/Pictures/my pic.jpg

then running pull.sh ~/Dekstop/RECOVERY filepaths.txt recovers the files to the following paths:

~/Desktop/RECOVERY/Users/jivan/Documents/my doc.pdf
~/Desktop/RECOVERY/Users/jivan/Pictures/my pic.jpg

I added the desired entries in filepaths.txt with some programmatic assistance, and was then able to successfully recover the vast majority of my files. For any particularly important files which this script fails to recover (due to bugs in the software I've written or additional malformed/missing APFS structures on the affected disk) I'll have to dig deeper, but this is effectively solved now.

All tools mentioned are available in the Git repo.

Source Link
Jivan Pal
  • 1.3k
  • 12
  • 21

Answering this question myself so that I can mark it as solved; see the update to the question made on 2019-11-14.