Take the 2-minute tour ×
Ask Different is a question and answer site for power users of Apple hardware and software. It's 100% free, no registration required.

I would like to call a command line utility in Mac OS X 10.8 that gives me the ability to convert a text file saved in standard Western Mac OS Roman encoding to the more generic UTF-8.

I will be calling the utility from an AppleScript that I have created. AppleScript is extremely slow when working with very large text blocks. As such, I want to do my text parsing and conversion using the OS X command line. I have found a tool called, "sed", which allows me to do the text parsing. However, there are still many elements of the file that need to be cleaned up, characters that appear as garbage if the file is opened as utf-8 (e.g. smart quotes and ellipses).

I am thinking that forcing a text encoding conversion may help to eliminate all non-utf8 characters in the file. However, I cannot see how "sed" can easily convert the text encoding.

I will have already saved the temp txt file, as MacRoman, to disk using the built-in AppleScript routines.

Do any of you have any ideas as to a built-in command-line tool that can convert text encoding? Command-line for performance and built-in, since other users of my script won't necessarily have the proper toolset if it's not built-in.

Thanks for your help!

share|improve this question

migrated from stackoverflow.com Feb 10 '13 at 16:31

This question came from our site for professional and enthusiast programmers.

    
The command is iconv. –  bmargulies Feb 10 '13 at 2:10
    
@bmargulies: When I try iconv on a file encoded in MacRoman and try to convert it into UTF-8, I get garbage characters in place of the original "unusual characters". For example, the ellipsis converts into "Äö√Ѭ∂‚Äö√Ѭ∂". I would expect the ellipsis to gracefully convert into six periods. Smart quotes are the same, they convert into weird text as well, turning into "Äö√Ñ√≤". The syntax that I use is: cat source.txt | iconv -f MacRoman -t UTF-8 > iconv_test.txt Is there a specific CLI syntax that would tell iconv to gracefully convert all text into appropriate replacements? –  Darkstar Feb 10 '13 at 4:40
    
And what arguments do you pass? –  bmargulies Feb 10 '13 at 13:25
    
@Darkstar--your same procedure worked fine for me. Are you sure your source was in MacRoman? –  Tom Gewecke Feb 10 '13 at 18:57
1  
Excel should not have this problem, you should be able to set the encoding for imports to whatever you want, e.g. see superuser.com/questions/280603/… –  Tom Gewecke Feb 13 '13 at 13:11

2 Answers 2

iconv is definitively the tool of choice here:

iconv -f MACROMAN -t UTF-8 your-roman-encoded-file.txt > utf-8-encoded-file.txt

Run iconv --list to see a list of all supported encodings.

share|improve this answer
    
once I figured out that it was not MacRoman, but was "iso-8859-1", I tried again. It still didn't do what I wanted it to do. I don't think iconv can do what I want it to do: gracefully replace all of the fancy extended characters with standard periods and apostrophes and double quotes. –  Darkstar Feb 11 '13 at 22:39

Another way to convert non-ASCII characters to ASCII variants is to use iconv -t ASCII//TRANSLIT:

$ echo ‘’“”–—…äé | iconv -t ASCII//TRANSLIT
''""--..."a'e

ASCII//IGNORE would remove non-ASCII characters, but you can also do that with for example tr -dc '\0-\177'.

share|improve this answer
    
Does iconv permit to transliterate: ä → a" ? –  daniel Azuelos Nov 12 '13 at 9:37
    
@danielAzuelos I don't know. The way characters are replaced depends on the implementation though: for example the iconv that comes with Debian replaces ä with just a. –  Lri Nov 12 '13 at 18:09

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.