3

I need to convert the character encoding in some text files created by a third-party app on my MBP Catalina 10.15.6. I'm in unfamiliar waters here, so please indulge my ignorance. Also, please note that the 3rd party app is not the subject of this question - understanding how to reconcile the different character sets used in macOS is the subject.

I use an application (LTspice) on my MBP occasionally. There is also a Windows version of LTspice. LTspice provides a GUI for creating a circuit schematic, and LTspice creates a plain text file (.asc extension) to encode the schematic and other directives and parameters created in the LTspice GUI; this is the file I need to convert.

I assumed the .asc files were not ASCII-encoded, and so I ran the file utility on the .asc file to learn how they were encoded:

% file -I '/Users/seamus/Documents/LTspice/Rounding demo-MacMod.asc'
/Users/seamus/Documents/LTspice/Rounding demo-MacMod.asc: application/octet-stream; charset=binary

binary?!... This made no sense to me. I can open and edit this file in TextEdit. All of the characters are recognizable ASCII characters - which I understand to be a subset of UTF-8.

My next step was to open the file in the BBedit app. This revealed new information. According to BBedit, the demo-MacMod.asc file reported by file -I as binary is actually: "UTF-16 Little Endian" format. I know this is confusing... In an effort to clarify, I've placed a couple of screenshots below to illustrate how this file is rendered in BBedit and TextEdit. The extra byte (¿) in the BBedit screenshot is a NUL.

I need a method (that I can automate/script) to convert these "UTF-16 Little Endian" files to "US-ASCII". I thought that the iconv tool would be perfect for this job:

iconv -l
...
# long list of character encodings which included:
US-ASCII
UCS-2LE UNICODELITTLE

UCS-2LE UNICODELITTLE looked like the best match to "UTF-16 Little Endian", but:

% iconv -f 'UCS-2LE UNICODELITTLE' -t 'US-ASCII' '/Users/seamus/Documents/LTspice/Rounding demo-MacMod.asc' > '/Users/seamus/Documents/LTspice/Rounding demo-MacMod-iconvASCII.asc'
iconv: conversion from UCS-2LE UNICODELITTLE unsupported
iconv: try 'iconv -l' to get the list of supported encodings

I don't know why I get this response. Clearly iconv -l says that UCS-2LE UNICODELITTLE is supported. Whether it's the correct match for "UTF-16 Little Endian" is another question, but I find nothing in the list that looks to be a better match.

So this is the gist of my question. I think it could be answered in one of two ways:

  1. What is my error in the use of iconv, or in my reading of man iconv or iconv -l?

  2. Is there another option for converting "UTF-16 Little Endian" to "US-ASCII" that can be automated/scripted?

SCREENSHOTS:

in TextEdit:

enter image description here

in BBedit:

enter image description here

1
  • It's more correct to describe UTF-8 as a superset on ASCII.
    – Pedro
    Dec 6, 2020 at 11:17

2 Answers 2

1

With BBEdit this is easy. First open the file in BBEdit. If you let BBEdit install its command line tool you can even do this from Terminal with bbedit /path/to/filename. If the file has opened as the wrong encoding, select File > Reopen Using Encoding > correct encoding. I think it would be worth trying reopening using UTF-16 Little-Endian & UTF-16 Little-Endian, no BOM to see if either of those has the file open as desired. When you have the file correctly opened, select File > Save As.... In the Save As dialog box you can choose the desired encoding, & also the line ending type if that matters.

For dealing with inverted red question marks, probably null (ASCII 0), select Text > Zap Gremlins... for the dialog below...

enter image description here

Using the options shown there should give a state like what you see in TextEdit. Try different options on copies of a couple of your files.

Because BBEdit has a command line tool, you should be able script it once you have the correct options. BBEdit also works with AppleScript & Automator.

You can download BBEdit for free from the link I've given. It will start in demo mode, & when the demo mode expires it will continue to run in free mode where the features you need are still available.

7
  • Thanks - I thought I made it clear in my Q that this can be done in BBedit, but the answer I need is one that I can script/automate: Is there another option for converting "UTF-16 Little Endian" to "US-ASCII" that can be automated/scripted?
    – Seamus
    Dec 6, 2020 at 10:23
  • Oops! I left out an entire portion of my intended answer. Look again in a few minutes.
    – Pedro
    Dec 6, 2020 at 10:28
  • 1
    I've got a BBedit license (used it for years & love it), so I'll look into the command-line tool. I have asked BBedit tech support about automation options though, and they didn't mention the cmd-line tool...
    – Seamus
    Dec 6, 2020 at 11:06
  • And don't overlook man bbedit to find out what the command line tool does for you.
    – Pedro
    Dec 6, 2020 at 11:11
  • Right, but man bbedit doesn't seem to suggest there is any support for changing encoding used in a file. The GUI certainly has it - "Zap Gremlins" & search & replace at least. I just don't see any way to invoke those using the cmdline ver of bbedit
    – Seamus
    Dec 6, 2020 at 11:19
0
  1. What is my error in the use of iconv, or in my reading of man iconv or iconv -l?

The error was simply mis-reading the output of iconv -l:

iconv -l
...
# long list of character encodings which included:
US-ASCII
UCS-2LE UNICODELITTLE

The character encoding is correctly read as UCS-2LE - not UCS-2LE UNICODELITTLE. Using that as the option in iconv:

% iconv -f 'UCS-2LE' -t 'US-ASCII' '/Users/seamus/Documents/LTspice/Rounding demo-MacMod.asc' > '/Users/seamus/Documents/LTspice/Rounding demo-MacMod-iconvASCII.asc'

Which gives the desired conversion from "UTF-16 Little Endian" to "US-ASCII"

As far as the question re file -I reporting this text file as a binary file - I have no clue at this time.

3
  • Binary as it has windows line endings and bytes not in 1-127 ie non ascII. So can't be read and the contents interpreted or at least does not meet a known byte sequence
    – mmmmmm
    Dec 7, 2020 at 15:55
  • @mmmmmm: Are you saying, "If it's not ASCII, then it's binary"? The TextEdt app opens and renders it perfecty - just as if it were an ASCII file.
    – Seamus
    Dec 7, 2020 at 23:28
  • Texted it probably uses a different algorithm to work out what the file is. Note it is in utf16 of some sort not ascii
    – mmmmmm
    Dec 8, 2020 at 8:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .