What the difference and usage of encodings UTF-8 and UTF-8-MAC in iconv?
I thought it was the difference between \n and \r(MAC OS 9) at first.
But I tried iconv -f UTF-8 -t UTF-8-MAC filename > filename2
The file content doesn't change in hex view.
-
It doesn't change in text view because the two version are (supposed to be) rendered the same.– WGroleauOct 1, 2022 at 18:56
-
@WGroleau I was talking about hex view not text view– maP1E bluEOct 2, 2022 at 19:04
-
Sorry—my eyes must be going. I was sure I saw "text view."– WGroleauOct 2, 2022 at 19:39
Add a comment
|
1 Answer
As explained here, utf-8-mac is the utf 8 version of a text after application of Unicode normalization NFD (e.g accented characters are represented by the base character plus a combining accent character), with certain codepoint ranges excluded from the decomposition operation.
For example character é can be represented in two different equally valid ways in Unicode:
- "\x{00E9}" - single codepoint, LATIN SMALL LETTER E WITH ACUTE, utf-8 C3 A9, "composed".
- "\x{0065}\x{0301}" - two codepoints, LATIN SMALL LETTER E and COMBINING ACUTE ACCENT, utf-8 65 CC 81, "decomposed"
UTF-8-MAC will ensure that the second, decomposed form is always used.
answered Dec 22, 2018 at 20:57
-
Note that iconv seems to consistently quit with "unable to convert" when it reaches somewhere between 3900 and 4100 bytes into the file. For several alternatives, see apple.stackexchange.com/questions/448408/…– WGroleauOct 1, 2022 at 18:55
-