I would like to call a command line utility in Mac OS X 10.8 that gives me the ability to convert a text file saved in standard Western Mac OS Roman encoding to the more generic UTF-8.

I will be calling the utility from an AppleScript that I have created. AppleScript is extremely slow when working with very large text blocks. As such, I want to do my text parsing and conversion using the OS X command line. I have found a tool called, "sed", which allows me to do the text parsing. However, there are still many elements of the file that need to be cleaned up, characters that appear as garbage if the file is opened as utf-8 (e.g. smart quotes and ellipses).

I am thinking that forcing a text encoding conversion may help to eliminate all non-utf8 characters in the file. However, I cannot see how "sed" can easily convert the text encoding.

I will have already saved the temp txt file, as MacRoman, to disk using the built-in AppleScript routines.

Do any of you have any ideas as to a built-in command-line tool that can convert text encoding? Command-line for performance and built-in, since other users of my script won't necessarily have the proper toolset if it's not built-in.

Thanks for your help!

  • The command is iconv. – bmargulies Feb 10 '13 at 2:10
  • @bmargulies: When I try iconv on a file encoded in MacRoman and try to convert it into UTF-8, I get garbage characters in place of the original "unusual characters". For example, the ellipsis converts into "Äö√Ѭ∂‚Äö√Ѭ∂". I would expect the ellipsis to gracefully convert into six periods. Smart quotes are the same, they convert into weird text as well, turning into "Äö√Ñ√≤". The syntax that I use is: cat source.txt | iconv -f MacRoman -t UTF-8 > iconv_test.txt Is there a specific CLI syntax that would tell iconv to gracefully convert all text into appropriate replacements? – Darkstar Feb 10 '13 at 4:40
  • And what arguments do you pass? – bmargulies Feb 10 '13 at 13:25
  • 1
    Is there some reason you must have ascii rather than the utf-8 which you originally requested? There is of course no ascii equivalent of many characters in MacRoman or ISO Latin 1, so I don't think you can do what you want with this kind utility. Search/replace is probably required. – Tom Gewecke Feb 12 '13 at 15:06
  • 1
    Excel should not have this problem, you should be able to set the encoding for imports to whatever you want, e.g. see superuser.com/questions/280603/… – Tom Gewecke Feb 13 '13 at 13:11

Another way to convert non-ASCII characters to ASCII variants is to use iconv -t ASCII//TRANSLIT:

$ echo ‘’“”–—…äé | iconv -t ASCII//TRANSLIT

ASCII//IGNORE would remove non-ASCII characters, but you can also do that with for example tr -dc '\0-\177'.

| improve this answer | |
  • Does iconv permit to transliterate: ä → a" ? – dan Nov 12 '13 at 9:37
  • @danielAzuelos I don't know. The way characters are replaced depends on the implementation though: for example the iconv that comes with Debian replaces ä with just a. – Lri Nov 12 '13 at 18:09

iconv is definitively the tool of choice here:

iconv -f MACROMAN -t UTF-8 your-roman-encoded-file.txt > utf-8-encoded-file.txt

Run iconv --list to see a list of all supported encodings.

| improve this answer | |
  • once I figured out that it was not MacRoman, but was "iso-8859-1", I tried again. It still didn't do what I wanted it to do. I don't think iconv can do what I want it to do: gracefully replace all of the fancy extended characters with standard periods and apostrophes and double quotes. – Darkstar Feb 11 '13 at 22:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .