1

I recently did a backup of a FAT32 disk which contained a lot of diverse data. It contains many media files, many big binary files, and as usual with FAT32, it has all kinds of odd (but potentially important) permissions for all of these different files. The whole directory of this backup weighs in at a few hundred GB. I would like to have a compressed archive of this backup, but I have a few requirements from whatever archival utility that I use:

  • It needs to preserve extended attributes
  • It needs to preserve timestamps
  • It needs to preserve permissions
  • I would like for it to preserve as much metadata as possible

I know that there are many different archiving utilities which fit at least some of my needs, but I would like to know of one which fits them all. I.e., which one can maintain the most metadata and also (preferably) has a good compression ratio for a diverse data set.

Improve this question
7
  • 1
    What's wrong with the simple Right click > Compress ?
    – Tetsujin
    Apr 24, 2018 at 7:59
  • FAT32 does not understand traditional unix permissions nor ACLs. What you are seeing in the Finder is kernel magic where all permission bits for the owner, group, and others are set and the owner of the files/folders is set to the _unknown user. Timestamps do not survive well between HFS+ and FAT32 file systems. Extended attributes would be in the form of AppleDouble format, where data is stored in one file and attributes in another file.
    – fd0
    Apr 24, 2018 at 19:32
  • @fd0 Thanks for the info. Does that mean that just about any compression / archival utility will lose timestamp metadata and potentially other kinds of metadata (such as extended attributes)?
    – GDP2
    Apr 24, 2018 at 20:11
  • @fd0 ACLs and every other piece of non-trivial metadata is stored in extended attributes, which is preserved in the AppleDouble (._ files). If you carry the dot files, you retain most metadata. Ownership is never set to unknown user when . files are present.
    – boris42
    Apr 24, 2018 at 20:30
  • @boris42 Hmm, thanks for the extra information. I'll have to take what both of you are saying and figure out how exactly it applies in my situation.
    – GDP2
    Apr 25, 2018 at 7:17

2 Answers 2

Reset to default
1

To preserve everything a Mac sees on the volume I'd suggest you make a compressed disk image with Disk Utility. Launch Disk Utility, select your external volume and select File/New Image from "Volume" or File/New Image from Folder and choose the compressed image format. It will definitely store everything and is the same underlying mechanism Time Machine uses when making a backup to a network destination.

Improve this answer
5
  • Disk Utility certainly doesn't seem to be a bad option for what I'm wanting. How good is the compression ratio? Also, what compression algorithm is used? I've used it before with fairly good results but I'd like to know how it compares to 7zip which is suggested in another answer.
    – GDP2
    Apr 24, 2018 at 20:08
  • @GDP2 Disk Utility (or hdiutil) use UDZO (zlib) and UDBZ (bzip2). I did some testing with a 300Mb folder with movies. 7zip made smallest file (223Mb), followed by UDBZ (263Mb), followed by zip (278Mb) followed by UDZO (282Mb).
    – boris42
    Apr 24, 2018 at 20:57
  • Thanks for the testing and statistics there. I'll have to mull it over a little more and see which one I think is best. I'll probably actually test both Disk Utility and 7zip and see whether the differences between them are significant.
    – GDP2
    Apr 25, 2018 at 7:19
  • Two things to consider - time to backup and how easy is to do a full or partial restore. In above testing 7z and UDBZ took about the same time (bit over 1 minute), zip and UDZO are also about the same (10-15 seconds). The extraction from 7z or zip for partial restore is either command-line tedious for requires 3rd party UI app, whereas you can mount a compressed disk image and use all of its contents as a volume immediately without the need to have free space for extracted files. For me this is a clear win. I've been using UDZO for years and UDBZ where space saved matters (over email and such)
    – boris42
    Apr 25, 2018 at 7:39
  • Thanks, Boris. I think that this option is the best for what I asked. Using Disk Utility or hdiutil seems to be the path of least resistance, although the compression ratio may be inferior. 7z requires a tarball in order to store metadata, which is not bad, but it feels slightly less reliable. Maybe it's just paranoia on my part, though.
    – GDP2
    May 3, 2018 at 6:58
1

If it's FAT32, use 7zip. Turn on the "slower but more effective" options, which use a larger dictionary. The results can be spectacular; I use it to move database backups around machines and I often get 96% compression.

Improve this answer
2
  • For OS X, p7zip seems to be the only option. That would be fine, except that the usage is a little complicated and I also see caveats in the man page such as this. Do you have any thoughts for the best usage of p7zip and whether these caveats are important for my situation?
    – GDP2
    Apr 24, 2018 at 20:07
  • I went ahead and accepted @boris42's answer just because it feels slightly more reliable than 7z. I think I may end up using 7z because of the better compression ratio, though. Yet, for what I asked, I think that Disk Utility / hdiutil seems to be a slightly more reliable backup strategy than a 7-zipped tarball. That may simply be paranoia on my part.
    – GDP2
    May 3, 2018 at 7:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .