Conversion script for Unicode combining characters?

Question

I have Greek text that used combining characters for all accented letters. Some process that I was using with it converted these automatically to the single character equivalent.  This causes a problem for me in that I need the two versions to match in another app.  Or at least to sort into the same order.

I can hunt for one at a time and use find/replace to fix that one, but this is rather tedious.

Is there a simple way to convert all of them? I can't do the original the way the second version was done, because they are different file types.

Answer 1

Here are some options to convert all decomposed Unicode characters into their precomposed form. I'll assume that the files are encoded in UTF-8.

iconv

iconv is a character set conversion tool (based on the iconv codeset conversion functions). On macOS, it supports the "UTF-8-MAC" encoding for decomposed UTF-8 characters (see What the difference and usage of encodings UTF-8 and UTF-8-MAC in iconv?).

Usage:

iconv -f UTF-8-MAC -t UTF-8 < decomposed.txt > composed.txt

I found that iconv has problems with characters outside of the basic multilingual plane (emojis, flags, ...). As an example, it can not convert the characters 𐂃 or 😀. The following solutions do not have that problem.

Perl

A simple Perl script (found here on Stack Overflow) can convert decomposed to precomposed UTF-8:

#!/usr/bin/perl

use Encode qw/decode_utf8 encode_utf8/;
use Unicode::Normalize;

while (<>) {
    print encode_utf8( NFC(decode_utf8 $_) );
}

Usage:

perl compose.pl < decomposed.txt > composed.txt   

Perl has also the -i option to modify a file “in situ”:

perl -i compose.pl file.txt 

uconv

(Found here on the Unix & Linux Stack Exchange.) uconv is not part of macOS, but can be installed from Homebrew (brew install icu4c) or Macports (port install icu).

Usage:

uconv -x any-nfc < decomposed.txt > composed.txt

Swift

If you have Xcode installed on your computer then a simple Swift script does the trick:

#!/usr/bin/swift

import Foundation

while let line = readLine() {
    print(line.precomposedStringWithCanonicalMapping)
}

Usage:

swift compose.swift < decomposed.txt > composed.txt

Answer 2

A MacOS app which can do these kinds of conversions is UnicodeChecker. It creates various Services that you can access when you select text in a document.

Answer 3

BBEdit has a 'Precompose Unicode' command in its Text menu. There is also an AppleScript command of the same name in the app's AS library.