I have an issue for which I have a work-around solution. I'm hoping to eliminate the work-around to improve production efficiency.
My associate's server automatically runs a script that outputs a large file of research data (all text) in a json
string and saves it as a huge (4MB+) .json
file.
I retrieve this file via ftp. My understanding is that the ftp software (Fetch) delivers the file to me just as it was saved.
When I try to process the file contents (from Terminal) using a Python script, I receive this error:
Traceback (most recent call last):
File "/Users/me/Documents/client drop folders/client1/pythonScript.py", line 61, in <module>
sourceData = json.loads(line.decode("utf-8"))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 2547902: invalid continuation byte
The character at this position is a space — though that seems moot, considering my work-around:
My work-around is simply opening the file and resaving it (either by using VSCode's standard save dialogue OR by simply using my Mac's TextEdit and choosing to save with utf-8
character encoding and a .json
extension) and overwriting the file.
The Python file processes the json with no errors after this re-saving of the source file.
If I run file -I sourceFile.json
on the save before re-saving it, then it produces:
sourceFile.json: text/plain; charset=utf-8
After the re-save, the same Terminal command produces the same:
sourceFile.json: text/plain; charset=utf-8
Why would the file perform differently before and after re-saving it?
I would appreciate any knowledgeable advice or answers. Is there a better troubleshooting method than the Terminal file -I
command?
Thanks in advance.
Appendices:
diff output:
1c1
< (entire content of file 1 from server)
\ No newline at end of file
---
> (entire content of file 2 saved on Mac)
\ No newline at end of file
cmp output:
old/file.json new/file.json differ: char 2547903, line 1
utf8_encode
Many thanks to both of you. You've steered me towards some excellent and new-to-me trouble-shooting tools here. If either of you want to post an answer, I'll swing by in a few days and approve, upvote, laud and magnify! Best.