Ask Different is a question and answer site for power users of Apple hardware and software. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I would like to call a command line utility in Mac OS X 10.8 that gives me the ability to convert a text file saved in standard Western Mac OS Roman encoding to the more generic UTF-8.

I will be calling the utility from an AppleScript that I have created. AppleScript is extremely slow when working with very large text blocks. As such, I want to do my text parsing and conversion using the OS X command line. I have found a tool called, "sed", which allows me to do the text parsing. However, there are still many elements of the file that need to be cleaned up, characters that appear as garbage if the file is opened as utf-8 (e.g. smart quotes and ellipses).

I am thinking that forcing a text encoding conversion may help to eliminate all non-utf8 characters in the file. However, I cannot see how "sed" can easily convert the text encoding.

I will have already saved the temp txt file, as MacRoman, to disk using the built-in AppleScript routines.

Do any of you have any ideas as to a built-in command-line tool that can convert text encoding? Command-line for performance and built-in, since other users of my script won't necessarily have the proper toolset if it's not built-in.

Thanks for your help!

share|improve this question

migrated from Feb 10 '13 at 16:31

This question came from our site for professional and enthusiast programmers.

The command is iconv. – bmargulies Feb 10 '13 at 2:10
@bmargulies: When I try iconv on a file encoded in MacRoman and try to convert it into UTF-8, I get garbage characters in place of the original "unusual characters". For example, the ellipsis converts into "Äö√Ѭ∂‚Äö√Ѭ∂". I would expect the ellipsis to gracefully convert into six periods. Smart quotes are the same, they convert into weird text as well, turning into "Äö√Ñ√≤". The syntax that I use is: cat source.txt | iconv -f MacRoman -t UTF-8 > iconv_test.txt Is there a specific CLI syntax that would tell iconv to gracefully convert all text into appropriate replacements? – Darkstar Feb 10 '13 at 4:40
And what arguments do you pass? – bmargulies Feb 10 '13 at 13:25
@Darkstar--your same procedure worked fine for me. Are you sure your source was in MacRoman? – Tom Gewecke Feb 10 '13 at 18:57
Excel should not have this problem, you should be able to set the encoding for imports to whatever you want, e.g. see… – Tom Gewecke Feb 13 '13 at 13:11
up vote 4 down vote accepted

Another way to convert non-ASCII characters to ASCII variants is to use iconv -t ASCII//TRANSLIT:

$ echo ‘’“”–—…äé | iconv -t ASCII//TRANSLIT

ASCII//IGNORE would remove non-ASCII characters, but you can also do that with for example tr -dc '\0-\177'.

share|improve this answer
Does iconv permit to transliterate: ä → a" ? – daniel Azuelos Nov 12 '13 at 9:37
@danielAzuelos I don't know. The way characters are replaced depends on the implementation though: for example the iconv that comes with Debian replaces ä with just a. – user495470 Nov 12 '13 at 18:09

iconv is definitively the tool of choice here:

iconv -f MACROMAN -t UTF-8 your-roman-encoded-file.txt > utf-8-encoded-file.txt

Run iconv --list to see a list of all supported encodings.

share|improve this answer
once I figured out that it was not MacRoman, but was "iso-8859-1", I tried again. It still didn't do what I wanted it to do. I don't think iconv can do what I want it to do: gracefully replace all of the fancy extended characters with standard periods and apostrophes and double quotes. – Darkstar Feb 11 '13 at 22:39

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.