2

I have an issue for which I have a work-around solution. I'm hoping to eliminate the work-around to improve production efficiency.

My associate's server automatically runs a script that outputs a large file of research data (all text) in a json string and saves it as a huge (4MB+) .json file.

I retrieve this file via ftp. My understanding is that the ftp software (Fetch) delivers the file to me just as it was saved.

When I try to process the file contents (from Terminal) using a Python script, I receive this error:

Traceback (most recent call last):
  File "/Users/me/Documents/client drop folders/client1/pythonScript.py", line 61, in <module>
    sourceData = json.loads(line.decode("utf-8"))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 2547902: invalid continuation byte

The character at this position is a space — though that seems moot, considering my work-around:

My work-around is simply opening the file and resaving it (either by using VSCode's standard save dialogue OR by simply using my Mac's TextEdit and choosing to save with utf-8 character encoding and a .json extension) and overwriting the file.

The Python file processes the json with no errors after this re-saving of the source file.

If I run file -I sourceFile.json on the save before re-saving it, then it produces:

sourceFile.json: text/plain; charset=utf-8

After the re-save, the same Terminal command produces the same:

sourceFile.json: text/plain; charset=utf-8

Why would the file perform differently before and after re-saving it?

I would appreciate any knowledgeable advice or answers. Is there a better troubleshooting method than the Terminal file -I command?

Thanks in advance.

Appendices:

diff output:

1c1
< (entire content of file 1 from server)
\ No newline at end of file
---
> (entire content of file 2 saved on Mac)
\ No newline at end of file

cmp output:

old/file.json new/file.json differ: char 2547903, line 1
Improve this question
10
  • 1
    0xc3 isn't a space, it's à [which is often one of those letters that appears randomly in badly-made movie subtitles;) Space is 0x20.
    – Tetsujin
    Mar 14, 2022 at 16:16
  • 2
    It'd be interesting to see a binary compare of the files before save and after save, but basically you're having the file make a round trip through some editor that is not binary-safe (VS Code certainly is not). So the file was interpreted, and then written out again.
    – Marc Wilson
    Mar 14, 2022 at 16:39
  • 2
    The cmp shows that it does change n VSCode. Bascially the original file is not valid utf-8 iwe need to see hex dumps around char 2547903 You need to look at the task generating the file.
    – mmmmmm
    Mar 15, 2022 at 17:10
  • 1
    @MarcWilson and mmmmmm You guys must think I'm a troubleshooting n00b! I was able to get the "builder" script from the source server. As mmmmmm predicted, the script missed out running the text through php's utf8 encoder: the simple function utf8_encode Many thanks to both of you. You've steered me towards some excellent and new-to-me trouble-shooting tools here. If either of you want to post an answer, I'll swing by in a few days and approve, upvote, laud and magnify! Best.
    – Parapluie
    Mar 21, 2022 at 16:50
  • 1
    @Tetsujin Haha! Well said, though I do feel like I should be throwing alms at the feet of all those who helped. Cheers; and thanks again.
    – Parapluie
    Mar 24, 2022 at 14:12

1 Answer 1

Reset to default
2

It appears that the issue was that the source file was indeed not saved with utf-8 encoding, despite what file -I indicated. I was able to examine the php script which produced the original json output and was also able to pass the text through the php function utf8_encode to solve the problem.

If others in this forum are having similar difficulties, I highly recommend reading through the comments on the opening question, as they provide good troubleshooting tools and advice.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .