0

How do I view the bytes of a filename as it’s stored an APFS volume (i.e. without going through Unicode normalization)?

Context: I migrated an old iTunes library to a new Music.app library, and ran the "Consolidate files" operation on the result to tidy things up. Then I used rsync and diff to compare the two on disk and got results like:

% diff -rq iTunes/iTunes\ Music Music/Media.localized/Music
...
Only in Music/Media.localized/Music: Múm
Only in iTunes/iTunes Music: Múm

This led me down the path of learning about Unicode normalization (explainers here and here). Those two strings have the same visual representation but are stored differently on disk. This comes up frequently with accented characters.

The question that arose was: How do I quickly get the on-disk byte representation of each name, so that I can be confident I understand what is going on?

BTW, for the original problem of comparing directories, I found this handy Python script (via this answer) which will consider Unicode-equivalent names as matching.

Improve this question

1 Answer 1

Reset to default
1

There has got to be a better way, but answering my own question as I at least found a way. Just copy the filename from Finder and paste it into your favorite hex dump tool.

% xxd
Múm
00000000: 4dc3 ba6d 0a                             M..m.
% xxd
Múm
00000000: 4d75 cc81 6d0a                           Mu..m.

Be careful though: If you accidentally get into Finder's rename box, it will overwrite the on-disk byte representation with a normalized form, leaving you very confused.

I lost a lot of time thinking that ls would print filenames as they are on disk, and I could pipe that to xxd or od, but as far as I can tell it actually normalizes before printing.

% ls -d Music/Media.localized/Music/M?m | tail -c 5 | xxd
00000000: 4dc3 ba6d 0a                             M..m.
% ls -d iTunes/iTunes\ Music/M?m | tail -c 5 | xxd
00000000: 4dc3 ba6d 0a                             M..m.

For a pure-CLI option, there's surely something more direct, but I did find this Perl script would do the job.

Improve this answer
3
  • 1
    The piping doesn't work because your shell expands the glob ? to single normalised characters, so whilst ? will match either/both UTF-8 encodings, c3 ba and 75 cc 81, it will always expand to the normalised one, c3 ba. The output of ls will then include that encoding rather than the 3-byte one, since ls always prints the particular input it received. You can see this behaviour of ls by providing the other byte string explicitly: ls "$(echo '4d 75cc81 6d' | xxd -r -p)" also matches the file, but piping that to xxd reveals that ls printed the 3-byte encoding instead.
    – Jivan Pal
    Commented Feb 7, 2022 at 16:52
  • Wow! Good start. Does mdls provide any useful information? If so, I might be able to help with the search / limitation syntax if you want to use that tool
    – bmike ♦
    Commented Sep 25, 2022 at 3:27
  • @bmike Not that I can see. That util is new to me, but I tried mdls -name kMDItemFSName -raw . | xxd from within both dirs, and got the same output.
    – duozmo
    Commented Sep 30, 2022 at 2:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .