2

I received a bunch of HTML files recently on a USB stick (FAT32), which were created on a Windows system, but with LF-line endings. I checked them in my browser (Firefox) on MacOS 10.14, and since on one of them I found a few changes to be done, I fired up my text editor and modified that file. After saving it, I tried to reload it, but Firefox refused now to show it and said that it has no permission to open this file.

I checked the file from the command line, and found that I could display the file using

less /Volume/Stick/original/x.html

fine and my changes were also present. I tried to restart Firefox, and even restarted the Mac itself, to no avail. Other browsers (Vivaldi, Chrome) also could not open the file. Looking at that directory /Volume/Stick/original using Finder, I saw that Finder could display a preview of all other files (y.html, z.html), but not of x.html I had modified.

Next I did a

ls -l /Volume/Stick/original

and found that all the "working" files (y, z, ...) showed their permissions as rwxrwxrwx (the unnecessary x-bit perhaps an artefact from the fact that the files were created on Windows), but the file x.html showed the permissions as rwxrwxrwx@. AFIK, the @ would indicate that the directory entry is in reality a link, but ls did not show where it would be linked to. Doing a stat of this file also did not provide more insight.

For experimenting a bit, I did a

mkdir /Volume/Stick/copy 
cp -v /Volume/Stick/original/* /Volume/Stick/copy 

thereby creating a copy of those files. A

ls -l /Volume/Stick/copy/x.html

also showed the @ in the permissions in the copied file. However the copied version, copy/x.html can be opened from any browser without problems, even the original/x.html can not. Even after redundantly copying back the broken file with

cp -v /Volume/Stick/copy/x.html /Volume/Stick/original

the version from original/x.html was unusable in the same way as before.

Of course I now do have a fully working version in /Volume/Stick/copy, but I still would like to understand what has happened here, and why editing the file ruined the file, although a copy of that edited file does not have any problems.

My best guess is that the problem is somehow related to the fact that FAT32 basically maintains files according to the infamous 8.3 naming scheme from MSDOS, and some shadow entry is used to store the "real" filename; but I don't see how this could explain the behaviour I experienced, because I never so far had problems when creating files on a USB stick on the Mac.

Any ideas, how to explain the mystery?

UPDATE:

Following the comment given by gidds, who pointed out the the @ does not indicate a symlink here, but an extended attribute, I applied a ls -l@ to both files (the non-working original, and the working copy). In both cases, the output was

com.apple.lastuseddate#PS     16

Probably that lastuseddate was set when I edited the file. I then did an xattr -p for this attribute, and both of them showed

69 FE DA 61 00 00 00 00 F0 9F C2 33 00 00 00 00

While I can't say whether this value is reasonable, but since the same value is shown in both files, it does not explain why one file "work" and one of them doesn't.

FOUND A FIX (but still can't understand it):

First, I found that restarting the Mac solved the problem in so far as the file can't be processed afterwards by all browsers, and also the preview in Finder looks OK. However, editing the file - or, for that matter, any other file on the USB stick - caused the problem to occur again.

After some experimentation, I found that the problem was tied to a particular text editor. I used the Cot Editor for editing the file, and with this, things get broken if the file is on a USB stick.

If, after rebooting, I edited the file using Tincta instead, everything is fine.

Further, I noticed that, once I got a broken file with the Cot Editor, I could also fix it by first copying the broken file on the command line to some other, temporary file, then erase the broken file, and finally copying it back from the temp file.

I don't know what Cot Editor is doing specially with those files, but at one point during my experiments, when I wanted to quit Cot Editor, I got a pop-up message saying: "The document x.html is on a volume that does not support permanent version storage". I have seen this message the first time and don't know, what it means, but at least it shows that there is something special about the USB volume which upsets this text editor.

8
  • 1
    I don't have the full answer, but @ after the file permissions in the output from ls -l indicates that the file has extended file attributes. (A symbolic link is shown by l at the start of the permissions and/or @ after the filename.) You can see what those extended attributes are with ls -l@, and manipulate them with xattr. macOS can store extended attributes on non-Mac filesystems (e.g. in hidden ._ files). It may be that Firefox has a problem with an extended attribute (such as a quarantine flag or file encoding).
    – gidds
    Jan 10 at 21:15
  • @gidds : Thank you for the hint. I don't think that Firefox in particular has a problem, because Vivaldi, Chrome and Finder had a problem as well. I will of course have a look at the extended attributes. Jan 11 at 5:50
  • @gidds : Checked with xattr. Does not look like this would explain the behaviour. Jan 12 at 14:10
  • Which version of the CotEditor are you using? Jan 19 at 15:55
  • Which version of Cot Editor are you running, and on which version of macOS? The Cot Editor's most recent GitHub commit seems to indicate that they're encountering some issues with saving, and it's possible that you're being affected by this or some other issue within that code (they've switched from a synchronous save to an asynchronous one, which may have resolved some type of race condition with the autosave feature). Jan 19 at 21:04

2 Answers 2

3

First of all, be aware that file permissions do not exist on FAT32 formatted drives. The permissions you see are defined by the way you mounted that drive (read more).

As you already found out by yourself, CotEditor is causing the trouble in your case. That's because CotEditor has an auto backup and versioning feature that is probably not compatible to the FAT32 file system:

Auto Backup

You don't need to lose your unsaved data anymore. CotEditor backups your documents automatically while editing.

From the code, you can read this about the Auto Save with Versions feature:

Automatically save documents to your file constantly while editing. This option also enables Versions, the modern system feature allowing you to go back to previous versions, as well as modifying document name or location from the window title bar.

Even if it turned off, CotEditor creates backup covertly for unexpected crash.

So unfortunately, you cannot disable the behaviour that causes you trouble. But you are not alone, and may be it gets fixed/changed in the future.

The problem with FAT32 might be related to this function which tries to write metadata, which is not supported in FAT32 file systems (except filename and timestamps). And this function will not work, because you don't have real file permissions here. You just have the file permissions of the mount, which cannot be altered dynamically.

4
  • I think your answer comes closest, but since I have already given away my bounty to a different answer, I would also like to award yours with a bounty. However I don't see an icon for creating a new bounty which I could award to you. Do you see a way how I can do this? Jan 19 at 8:03
  • Update: I found a way how to do it, but I need to wait 23 hours until I can award you the bounty. Thanks anyway. Jan 19 at 8:21
  • That would not have been necessary, but thank you anyway. If you are interested in CotEditor, you could bring this up as an issue on their GitHub. Jan 19 at 8:33
  • I already contacted the CotEditor developer by mail, whether a bug report makes sense. My impression is that the editor is not being actively maintained anymore, and did not want to create a detailed bug report unnecessarily. Jan 20 at 7:35
0
+300

My guess is that the issue is being caused by the Cot Editor not releasing whatever type of file lock it's establishing on the files that it opens from the FAT32-formatted volume.

Generally, when a program opens a file with exclusive access (i.e., when it locks the file), it should also release that lock after the file is closed (or saved) - which probably isn't happening in this scenario.

Since restarting the computer resolves the issue, a temporary file lock not getting released seems to be the most logical reason for this occurring (since all file locks are released when the system starts up). Additionally, a copy of that file wouldn't have a file lock held on it (since it's a copy, and not the file that was being edited by Cot Editor), which also points at some temporary file lock.

I'm not sure of a way to display a list of all files that have a file lock held within current versions of macOS. It may be possible to use lsof to see some of the processes that are locking files (e.g., sudo lsof | grep 'x\.html'). However, this may not work in all situations (e.g., like the one where the Cot Editor process is no longer running but hasn't released the lock). I believe that file locking information is held within kernel-space memory, and that no other processes have access to it. More info about how file locking is implemented in macOS might be available on the XNU kernel's GitHub page for fcntl.

Alternately: as you mention, the @ character showing in the permissions listing in Terminal indicates that the file has extended attributes set. It's also possible that one of those attributes is the system immutable (schg) or user immutable (uchg) bit. Using the chflags command (e.g., chflags nouchg x.html or chflags noschg x.html) might allow you 'unlock' the files that Cot Editor has left un-mutable (this probably isn't going to help based on the reasoning suggested above).

If other apps are able to edit those files normally, it's possible that there is some kind of FAT32 file editing bug that exists with the Cot Editor app (i.e., if it works normally on volumes formatted with APFS or HFS+?), which the developer of the software might be able to rectify.

4
  • Great explanation, but I feel that it does not explain the following: When I copied the presumably locked file, the copy itself worked obviously fine. This means that the lock, what ever it is, can only be a write lock, not a read lock (otherwise copying the file would also be rejected). This also explains why I could display the file on the command line (using less). But if the lock does not forbid reading, it does not explain why rendering the file (using a web browser) did not work. So, your last explanation (FAT32 edit bug) seems to me more plausible. Jan 19 at 7:52
  • 1
    There are several ways to open a file, and some of those file descriptor open options will allow a file to be read while leaving it in an unwritable state (e.g., O_SHLOCK vs. O_EXLOCK?). Here's a list of the different ways that Swift implements standard C file locks. Here's an explanation of the difference between those two lock types. I'm unsure which option CotEditor is using from a cursory glance. Jan 19 at 16:21
  • I understand this, but I think the problem is not an unwritable state here. A web browser hopefully does not want to open the file to be rendered for writing. Even if one browser would do this due to bad programming, it is highly unlikely that all browsers I tried behave the same in this way. An exclusive lock could explain it, but can it explain that the file can be displayed by less or cat from the command line? Do these utilities ignore the lock? Jan 20 at 7:32
  • I agree that it seems illogical for a browser to open a local file as O_RDWR, but it's definitely possible (the source for both FireFox and Chrome seem to indicate that they have those file handle open options available). I'm not sure how those open options can be used when opening a file, but regardless: it is possible to open files in different modes (O_RDONLY, O_WRONLY, O_RDWR). Since less and cat appear to use O_RDONLY by default, some type of write lock seems feasible. Jan 20 at 16:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .