2

I have Greek text that used combining characters for all accented letters. Some process that I was using with it converted these automatically to the single character equivalent.  This causes a problem for me in that I need the two versions to match in another app.  Or at least to sort into the same order.

I can hunt for one at a time and use find/replace to fix that one, but this is rather tedious.

Is there a simple way to convert all of them? I can't do the original the way the second version was done, because they are different file types.

9
  • iconv -f UTF-8 -t UTF-8-MAC filename > filename2 should do the trick, see apple.stackexchange.com/q/346453/30895
    – Martin R
    Commented Sep 30, 2022 at 5:03
  • Looks like that will work, thanks. (Though going the other way is better, because the actual filenames are composed form.)
    – WGroleau
    Commented Sep 30, 2022 at 6:27
  • So you want to convert from decomposed to composed form?
    – Martin R
    Commented Sep 30, 2022 at 11:01
  • Either way, but composed is better. iconv definitely can do either, but it seems to have a buffering problem—in large files, it stops and complains somewhere between 3950 & 4020 bytes. But when the alleged "unable to convert line" is near the beginning of the file, it is converted with no problem. A nine-hundred-line file had to be done in half-a-dozen sections. Looks like you have the answer.
    – WGroleau
    Commented Oct 1, 2022 at 4:01
  • I'd be surprised if BBEdit couldn't handle this.
    – benwiggy
    Commented Oct 1, 2022 at 15:15

3 Answers 3

4

Here are some options to convert all decomposed Unicode characters into their precomposed form. I'll assume that the files are encoded in UTF-8.

iconv

iconv is a character set conversion tool (based on the iconv codeset conversion functions). On macOS, it supports the "UTF-8-MAC" encoding for decomposed UTF-8 characters (see What the difference and usage of encodings UTF-8 and UTF-8-MAC in iconv?).

Usage:

iconv -f UTF-8-MAC -t UTF-8 < decomposed.txt > composed.txt

I found that iconv has problems with characters outside of the basic multilingual plane (emojis, flags, ...). As an example, it can not convert the characters 𐂃 or 😀. The following solutions do not have that problem.

Perl

A simple Perl script (found here on Stack Overflow) can convert decomposed to precomposed UTF-8:

#!/usr/bin/perl

use Encode qw/decode_utf8 encode_utf8/;
use Unicode::Normalize;

while (<>) {
    print encode_utf8( NFC(decode_utf8 $_) );
}

Usage:

perl compose.pl < decomposed.txt > composed.txt   

Perl has also the -i option to modify a file “in situ”:

perl -i compose.pl file.txt 

uconv

(Found here on the Unix & Linux Stack Exchange.) uconv is not part of macOS, but can be installed from Homebrew (brew install icu4c) or Macports (port install icu).

Usage:

uconv -x any-nfc < decomposed.txt > composed.txt

Swift

If you have Xcode installed on your computer then a simple Swift script does the trick:

#!/usr/bin/swift

import Foundation

while let line = readLine() {
    print(line.precomposedStringWithCanonicalMapping)
}

Usage:

swift compose.swift < decomposed.txt > composed.txt
1
  • Thanks. Immediate issue solved, but I'll look at all the others (including Tom Gewecke’s suggestion). Beats feeding perl a script containing a substitution for every possible combination!
    – WGroleau
    Commented Oct 1, 2022 at 14:56
2

A MacOS app which can do these kinds of conversions is UnicodeChecker. It creates various Services that you can access when you select text in a document.

1
  • I may already have that. If not, I'll get it. The MacOS-include iconv does the job if the file is small. (And already did for this incident.)
    – WGroleau
    Commented Oct 1, 2022 at 14:53
0

BBEdit has a 'Precompose Unicode' command in its Text menu. There is also an AppleScript command of the same name in the app's AS library.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .