0

I have several thousand text files, some of which contain invalid UTF-8 characters. I want to recursively remove all invalid characters from these files in place.

I am aware that many similar questions have been asked before, such as: how to remove non UTF-8 characters from text file but I have not found one that is both recursive and operates in place.

4

The great thing about UNIX commands is that you can combine them together. iconv doesn't know how to recurse into directories, but find does. It can call iconv on every file it sees.

(These commands will convert all files in the current directory and all directories within. Make sure you are in the directory you want to convert all files in recursively.)

To change all files with the extension .txt:

find . -type f -name '*.txt' -print0 | 
    while IFS= read -r -d $'\0' filename; do 
        iconv -f utf-8 -t utf-8 -c "$filename" > "$filename".iconv_cleaned_utf8
        mv "$filename".iconv_cleaned_utf8 "$filename"
    done

I suppose this code requires some explanation. What it does is:

  • find prints out all the filenames of the files involved, separated by a null byte (the null byte is the only invalid character for a file path)
  • bash reads the filenames and loops through them
  • iconv converts the file to a tempfile with an extra extension
  • we mv the tempfile to replace the original file.

If they have different extensions (this is for any and all files under the current directory), remove the -name *.txt

It's a bit cleaner if you have the sponge utility from moreutils, but that is not installed by default.

find . -type f -name '*.txt' -print0 | 
    while IFS= read -r -d $'\0' filename; do 
        iconv -f utf-8 -t utf-8 -c "$filename" | sponge "$filename"
    done
  • Thanks @0942v8653, I ran your command, there was lots of output to the terminal, however there were no edits to files I know to contain illegal characters. – pingu Apr 9 '16 at 10:18
  • Ooh, sorry, I didn't actually test it yet, it turns out iconv writes to the terminal. Will update my answer. – 0942v8653 Apr 9 '16 at 13:00
  • @pingu: edited. – 0942v8653 Apr 9 '16 at 13:43
0

This may not answer the whole question but I have been a happy user of BBEdit for quite a number of years (Pre-OSX at least...). I use it for HTML hacking and occasional opening of files that I can't figure out.

It has this nice feature called Zap Gremlins, which basically removes a whole bunch of strange and non-ASCII characters from any file. Which sounds a lot like what you need to do.

AND it is massively scriptable, but not free. It may be worth investigating to see if it will do what you want.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .