19

I would like to call a command line utility in Mac OS X 10.8 that gives me the ability to convert a text file saved in standard Western Mac OS Roman encoding to the more generic UTF-8.

I will be calling the utility from an AppleScript that I have created. AppleScript is extremely slow when working with very large text blocks. As such, I want to do my text parsing and conversion using the OS X command line. I have found a tool called, "sed", which allows me to do the text parsing. However, there are still many elements of the file that need to be cleaned up, characters that appear as garbage if the file is opened as utf-8 (e.g. smart quotes and ellipses).

I am thinking that forcing a text encoding conversion may help to eliminate all non-utf8 characters in the file. However, I cannot see how "sed" can easily convert the text encoding.

I will have already saved the temp txt file, as MacRoman, to disk using the built-in AppleScript routines.

Requirements:

  • Command-line for performance
  • Prefer native tools since other users of my script won't necessarily have the proper toolset if it's not built-in. (Although I could add a check to my script and abort if a needed tool isn’t present)

Do any of you have any ideas as to a built-in command-line tool that can convert text encoding or an existing package that is superior for this task?

Improve this question
6
  • @bmargulies: Actually, what I'm really trying to get is a means to automatically convert fancy characters, like ellipsis and smart quotes into "real", "pure" ASCII quotes, apostrophes, and periods. Can iconv do that? Or do I have to manually convert the files?
    – Darkstar
    Feb 11, 2013 at 22:36
  • @TomGewecke: Actually the source was not MacRoman, it was "iso-8859-1". However, using iconv didn't gracefully (and automatically) replace the fancy quotes and ellipsis with standard ASCII quotes and periods.
    – Darkstar
    Feb 11, 2013 at 22:37
  • 1
    Is there some reason you must have ascii rather than the utf-8 which you originally requested? There is of course no ascii equivalent of many characters in MacRoman or ISO Latin 1, so I don't think you can do what you want with this kind utility. Search/replace is probably required.
    – Tom Gewecke
    Feb 12, 2013 at 15:06
  • @TomGewecke: The reason I'm trying to simplify things is because the CSV files my AppleScript is creating look ugly in Excel due to the presence of these non-ASCII characters. Microsoft Excel (both Windows and on the Mac) is the only program that has problems with importing the CSV files that are encoded in iso-8859-1.
    – Darkstar
    Feb 13, 2013 at 5:26
  • 1
    Excel should not have this problem, you should be able to set the encoding for imports to whatever you want, e.g. see superuser.com/questions/280603/…
    – Tom Gewecke
    Feb 13, 2013 at 13:11

2 Answers 2

Reset to default
12

Another way to convert non-ASCII characters to ASCII variants is to use iconv -t ASCII//TRANSLIT:

$ echo ‘’“”–—…äé | iconv -t ASCII//TRANSLIT
''""--..."a'e

ASCII//IGNORE would remove non-ASCII characters, but you can also do that with for example tr -dc '\0-\177'.

Improve this answer
2
  • Does iconv permit to transliterate: ä → a" ?
    – dan
    Nov 12, 2013 at 9:37
  • @danielAzuelos I don't know. The way characters are replaced depends on the implementation though: for example the iconv that comes with Debian replaces ä with just a.
    – Lri
    Nov 12, 2013 at 18:09
34

iconv is definitively the tool of choice here:

iconv -f MACROMAN -t UTF-8 your-roman-encoded-file.txt > utf-8-encoded-file.txt

Run iconv --list to see a list of all supported encodings.

Improve this answer
1
  • once I figured out that it was not MacRoman, but was "iso-8859-1", I tried again. It still didn't do what I wanted it to do. I don't think iconv can do what I want it to do: gracefully replace all of the fancy extended characters with standard periods and apostrophes and double quotes.
    – Darkstar
    Feb 11, 2013 at 22:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .