2

Context:

I've been backing up my MacBook Pro with rysnc (now version 3.2.3, protocol version 31) to external hard drives formatted with NTFS. I'd like to continue with this procedure if possible.

Usually, for a given macOS version, I would use the StdExclusions file located at /System/Library/CoreServices/backupd.bundle/Contents/Resources/StdExclusions.plist
as a starting point to decide what files/folders to exclude.

In Big Sur, this StdExclusions file doesn't seem to exist (at least not in its usual location).

Sub-questions:

  1. In Big Sur, is there a new StdExclusions file or equivalent in another location?

  2. Given the new file system (APFS with firmlinks/wormholes and things that are still mysterious to me) should I exclude /System/Volumes/Data/Users and copy /Users or the other way around, or do I need to copy both?

  3. More generally, (and more importantly and to the point than sub-question 2) how should one approach the /System/Volumes/Data directory in context with the whole system when backing up?

Context detail:

Initially when I was using rsync version 2.6.9 protocol version 29, I didn't need to include the --ignore-errors flag but with rsync 3 I found I had to add that flag, so my fundamental rsync command takes this form:

copyall() {
  rsync -aAXPvh --modify-window=1 --partial-dir=../rsync_TMP_PARTIAL --delete-after --ignore-errors "$@"
}
BDL="--backup-dir=../DeletedFiles_$(date +\%Y-\%m-\%d)"
EXC="--exclude-from=/Location/of/my/modified/custom/exclusions/file"
copyall "${EXC}" --backup "${BDL}" "${SRC_BASE}/" "${DST_BASE}"
9
  • 1
    1) Exclude everything you don't need for your disaster recovery plan! How and what do you intend to recover? 2) NTFS is not the way to go - requires additional (non-Apple) software. 3) Do yourself a favour and use Time Machine.
    – Gilby
    Aug 6, 2021 at 22:20
  • @Gilby, for recovery, I'd be okay with having only a copy of my unique material (and config files saved that make re-installations easy); if trouble happens, I could re-install the OS if necessary and then copy over material from my backup drive and do any remaining re-installs. I believe I chose NTFS for flexibility in being able to access my backup drives from a variety of operating systems. A few years ago I bought software (Paragon NTFS for Mac 15) that allows me to use NTFS. On their website, looks like it supports Big Sur. The question about /System/Volumes/Data/Users is important to me. Aug 6, 2021 at 23:20
  • One of my mistakes: looks like the -E option changed to -AX (ACL & extended attributes) between rsync version 2 and 3, so rsync -aEPvh should instead be rsync -aAXPvh. And it looks like Paragon NTFS does support extended attributes. Does this cover all APFS metadata? I don't know. Aug 7, 2021 at 1:58
  • 1
    You only need to backup /System/Volumes/Data, the OS itself can‘t be restored anyway and is easily reinstalled from Recovery.
    – nohillside
    Aug 7, 2021 at 5:23
  • @nohillside, thanks. Could you explain how you came to this knowledge? And is there any official documentation that would confirm your knowledge? If I take your advice, it seems that the old StdExclusions file could be applied to this directory; so /.Spotlight-V100 would become /System/Volumes/Data/.Spotlight-V100 and similar for /.fseventsd, etc. I wonder though if there are new files I should be excluding particular to Big Sur. That is, what should be the standard exclusions for Big Sur? Aug 7, 2021 at 5:42

2 Answers 2

1

Using rsync to copy files to an NTFS drive will not provide a seamless recovery.

  • File and Folder metadata will be lost along with aliases, symlinks, etc.
  • Recovering applications, settings and state information will be largely lost and would require careful copying of files stored in /Library and ~/Library.
  • Any use of iCloud would need to be reestablished.
  • I am doubtful that you would able to recover the libraries used by the Photos and Music libraries to the state they were before disaster.

The best you will be able to do is to recover files in folders such as ~/Documents.

The standard exclusions used by Time Machine as that would leave you rsync-ing lots of files that would not provide recovery to the original state of the Mac.

Rather that looking for exclusions, I suggest that you need to first look at inclusions and determine what, potentially, can be recovered.

If using rsync you should limit your expectations to recovery of most data files, but without much of the state information stored in file metadata and /Library and ~/Library.

Regarding your second sub-question: /System/Volumes/Data/Users and /Users are the same thing.

Strong recommendation:

  • For a seamless disaster recovery, you need to use a backup program which is aware of macOS data structures. That would be Time Machine or third-party products like Carbon Copy Cloner or Chronosync.
5
  • I appreciate your pointers and recommendations; in the long-run I'll consider options you point to. In the short-run, since I don't require seemless recovery, I'll continue with rsync. If I include the options I mentioned in another comment on the main post (in particular -aAX, which includes extended attributes), does that get all the essential APFS metadata? I think symlinks are copying fine, and I don't think I need aliases. I like your emphasis on inclusions rather than exclusions, because I'll want to minimize what I'm copying. I guess I'll exclude /System/Volumes/Data/Users. Aug 7, 2021 at 2:08
  • @zeroparallax I am sure you know more, form the practical point of view, about rsync than I do. But certainly, including extended attributes will get the essential metadata at the file level. You may want to consider whether that will maintain file creation and modification dates (if that is important to you). But it really is about testing again and again until you are confident of good enough recovery.
    – Gilby
    Aug 7, 2021 at 2:57
  • I appreciate your answer and help, but I'd prefer to mark as "accepted" an answer that is more definitive about solving the "exclusions" issue for a general backup, in particular with reference to what the StdExclusions.plist equivalent might be for Big Sur, and also how one should approach the /System/Volumes/Data/ directory. Although I didn't highlight that in my question, I'm still confused about that. (Regarding your helpful focus on "inclusions", perhaps that could be another question about what is necessary in a minimalist backup.) Aug 7, 2021 at 3:46
  • I have no problem with you waiting for an answer more focused on your precise need. /System/Volumes/Data is the read-write part of the macOS boot disk. So '/System/Volumes/Data/Users` is the same as /Users since all of /Users is on the read-write part of boot disk. But /System/Volumes/Applications just has apps not part of macOS and so is different to /Applications which includes both read-only apps (like Mail) as well as added apps.
    – Gilby
    Aug 7, 2021 at 5:56
  • I disagree. Modern versions of rsync are perfectly capable of "lossless" backup and restoration. This is easy enough to demonstrate; see this for example.
    – Seamus
    Aug 8, 2022 at 7:05
1

The file system structure got rather elaborated with Big Sur (Big Sur boot volume layout gives a good overview). The positive part of it is that the content worth backuping up is all part of /System/Volumes/Data.

$ cd /System/Volumes/Data
$ ll
total 2
drwxrwxr-x  170 root  admin  5440 Aug  6 19:42 Applications/
drwxrwxr-x    5 root  admin   160 Dec 15  2020 Incompatible Software/
drwxr-xr-x   84 root  wheel  2688 Jul 26 20:30 Library/
drwxr-xr-x@   3 root  wheel    96 Jan  1  2020 System/
drwxr-xr-x   12 root  admin   384 Jan  1  2020 Users/
drwxr-xr-x   10 root  wheel   320 Aug  7 08:06 Volumes/
drwxr-xr-x    2 root  wheel    64 Oct 25  2020 cores/
dr-xr-xr-x    2 root  wheel     1 Aug  7 08:11 home/
drwxr-xr-x    2 root  wheel    64 Oct 25  2020 mnt/
drwxr-xr-x    5 root  wheel   160 Dec  3  2020 opt/
drwxr-xr-x    6 root  wheel   192 Jan  1  2020 private/
drwxr-xr-x    2 root  wheel    64 Oct 25  2020 sw/
drwxr-xr-x@   5 root  wheel   160 Jan  1  2020 usr/

(you may not have ./sw)

Looking at that list it seems save to skip

Incompatible Software/
Library/Caches/
Volumes/
cores/
home/
mnt/

in general.

On user level, the things which can be skipped are ~/Library/Caches and any VMs you have laying around. Whether on wants to skip ~/Downloads is more a matter of personal taste (TimeMachine skips as far as I remember).

PS: In general I would recommend against backup up macOS data on an NTFS formatted volume, there are too many things which can go wrong and you have to rely on 3rd party software to recover data afterwards). If NTFS is all you have, creating a sparsebundle DMG formatted with APFS and using this as a backup target might be better.

1
  • TM does not skip ~/Downloads. Will the OP be able to correctly (usably) recover much of ~/Library from an NTFS backup - I am thinking of things like ~/Library/Containers?
    – Gilby
    Aug 7, 2021 at 9:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .