1

I have a Time Machine created sparsebundle (HFS+) that reports the following limits when using hdiutil resize -limits:

 min     cur     max 
403186088   403186088   34359738368

That's a current size of 206 GB and a max of 17.5 TB. I draw the conclusion that I don't need to hdiutil resize the sparsebundle to allow it to expand. The host volume has an additional 200 GB free, so the bundle should therefore be able to expand about 100% -- however:

  • Disk utility reports free space as 1 GB
  • Time Machine reports free space as 1 GB

I expected them to detect 200 GB of free space available on the host volume. This is a problem since Time Machine will start deleting backups prematurely.

How do I make the sparsebundle expand when necessary to fill available space on the host file system?

Some additional questions that came up that could help to narrow it down:

  1. Which part of the system is responsible for expanding the bundle when required? Does Time Machine do this explicitly or is it done transparently underneath it? I.e. is this a problem with Time Machine or with something more fundamental?
  2. Is there a more reliable way to check whether Time Machine has/will run out of space than to look at the "free space" figure in Time Machine settings? I.e. can the numbers be trusted? I did remove and add the bundle as a backup destination with tmutil setdestination before checking free space.
  3. If Time Machine is explicitly responsible for expanding a sparsebundle used as a backup target, will it only do this when it itself mounts the sparsebundle as a step of a networked backup? In other words, when taking the networked drive and plugging it in locally with USB3, and then manually mounting the sparsebundle, would Time Machine just skip the step where it usually expands the sparsebundle target since the bundle is now considered local storage (which doesn't generally use sparsebundles)?

MacOS Mojave 10.14.6


Update

There are several subtle reasons that combine into why I think my sparsebundle doesn't expand automatically, and why I form the null hypothesis that the free space reported by Time Machine is not erroneous. They include:

  • I have no good reason to assume that the reported number in Time Machine and Disk Utility UI is erroneous. Conversely, I have been given no explicit assurances that the target volume will expand, nor to what extent.
  • Time Machine has to my recollection failed previously to expand the sparsebundle target that it was backing up to, despite lots of available space on the host volume.
  • Apple's track record of design suggests to me that if they only expose a single figure (remaining space available) in the Time Machine UI for me to base my decisions on with regards to what actions TM will take in turn, that it would've been removed already if it were an unreliable figure. I don't count the use of sparsebundles (i.e. networked backup) as such a small corner case as to warrant that level of unreliability to be ignored in their UI design phase.
  • When the drive is used as a network target, TM seems to report ~200/400 GB available, while attached as a local backup target it will report ~200/200 GB available. A locally mounted sparsebundle is also not showing up in the "add target" UI in TM, but has to be added using tmutil. This suggest an inconsistency in how backup targets are handled and that Time Machine manages sparsebundles explicitly.

I realize this is not conclusive, so I tried it out:

  1. I mounted the host volume over USB, and proceeded to resize the sparsebundle with hdiutil to be as small as I could make it, so that it appeared to have ~20 GB free according to all system GUI tools. It shrank by over 100 GB in that regard, so plenty of space to spare on the host volume.
  2. I mounted the bundle as a local volume, and proceeded to copy an 11 GB Xcode.xip file to it twice.
  3. The first transfer succeeded.
  4. The second transfer failed before even starting -- "not enough free space". 1.54 GB of additional space was required. This was expected.
  5. Conclusion: the sparsebundle did not expand transparently, despite over 100 GB of free space on the host volume. There is no reason for me to believe that the figure that Time Machine reports is erroneous, nor that the volume will expand transparently during TM operation.
6
  • How exactly did you resize the sparsebundle? - What are your min,cur,max values now? Sounds like you did something you did not want. – jksoegaard Apr 2 at 11:10
  • min, cur, max: 528340448 547356672 34359738368 -- I have to add that I did expand the volume to 400 GB, did a backup, and then shrank it again in order to perform the test, but the max value is unchanged. – Andreas Apr 2 at 11:12
  • So this means that you have stored 252 GB and you have set a limit of 267 GB saying you never want the system to automatically expand the size to more than 267 GB in total. – jksoegaard Apr 2 at 11:17
  • @jksoegaard Yes, that is the only conclusion I could draw in my original comment to your answer. However, I got the impression from your subsequent comments that you did not agree with my conclusion of the meaning of cur, where I determined that it should reasonably correspond to what you referred to as maximum capacity. From what you're saying now, I get the impression that you suggest that cur indeed is a max beyond which it will not expand. – Andreas Apr 2 at 11:22
  • It was really hard to understand what you meant in your comments. I have extended my answer to show you actually what is what. You need to distinguish between the arbitrary limit you set on the sparse bundle, and the amount of free space on your physical drive. – jksoegaard Apr 2 at 11:27
2

Regarding your first question: It is the operating system that automatically resizes sparse bundles as necessary when you add data to them. This is done transparently underneath Time Machine, so it is not something that Time Machine explicitly handles.

The sparse bundle itself can have a maximum capacity that exceeds the available space on your drive, as it does not actually occupy any space before you write data to it. You can change resize this, but it doesn't actually change anything about how much space it takes up on the physical drive, but rather just imposes a limit on how much can eventually be written to it.

Even though Disk Utility reports it as having no real free space, it will automatically expand to fill up the drive. When the physical drive is filled up, it is ofcourse not possible to add more to the sparse bundle until you clear up space or move the bundle to a larger drive.

When you run hdiutil resize -limits my.sparsebundle you'll get a min, cur and max value. You can use the following calculations to get "human-readable" numbers for them:

  min * 512 / 1024 / 1024 = MB data stored in the sparsebundle

  cur * 512 / 1024 / 1024 = limit in MB of how much data can be added

  max * 512 / 1024 / 1024 = the largest limit in MB you can set

You can change the limit at any time using:

hdiutil resize -size 500MB my.sparsebundle

This would change the limit to 500 MB. Obviously it doesn't make sense to set the limit lower than the current amount of data stored. Perhaps a bit counter intuitive, the limit can be set higher than the amount of disk space you actually have available. The limit cannot however be set higher than the max value, which in your case is 16 TB. The reason for the 16 TB is that you have it setup with a block size of 4 kB. You can create sparse bundles with a larger block size to get a higher max value up to the theoretical maximum of just shy of 8 exabytes.

Note that this limit does not have anything to do with how much disk space is actually taken up on your physical disk. You can find this number running for example:

du -sh my.sparsebundle

If for example use Disk Utility to create an empty sparsebundle with a size of 500 MB formatted with APFS, you'll see that the above command tells you that it takes up only approx. 13 MB on the physical disk.

When you mount the sparsebundle and add files to it, the amount of space taken up on the physical disk increases. I.e. the operating system transparently expands the sparsebundle structure on the physical disk to accommodate the extra data you add to it. However, it won't add more than the arbitrary limit, you have set it.

If you select "Info" on the mounted sparsebundle in Finder, it will show you how much data has been added to the sparsebundle as well as the amount of "Free" space. This amount of "Free" space is not the amount of free space on your physical drive, but rather how much is left until you hit the (artificial) limit set on your sparsebundle, which you can change at any time later.

8
  • Thanks for confirming. Since Time Machine sees only 1 GB free in this instance, and the expansion is transparent to it, I'm assuming that it will then conclude that old backups need to be deleted before it will try to write anything, suggesting that the bundle is at max capacity already — what other indication could an application like Time Machine possibly have for making that decision? The cur field of hdiutil resize -limits output should then reasonably correspond to the maximum capacity that you mention, making the max field a mystery to me a.t.m. – Andreas Apr 1 at 21:13
  • Would you say that my conclusion is wrong that e.g. Time Machine will look at the limited space remaining (1 GB) and make its decisions based on that? In the case of Time Machine this number is the only indication I can find of whether or not it considers the target volume full, so if that's not correct with regards to sparsebundles I'm a bit confused as to how to determine what TM actually considers the available space to be 🤔 – Andreas Apr 1 at 21:27
  • You have written nothing about WHY you think that Time Machine erraneously makes decisions based on a false sense of free disk space. Do you have observations that this is occuring? - It sounds a bit weird that it would be malfunctioning just for you without some kind of justification. – jksoegaard Apr 2 at 7:23
  • I think you're putting too much weight to the "min, cur, max" values. You seem to have misunderstood what they mean. Remember that hdiutil can output these values for several types of image files - not only sparse bundles. For some image formats it makes total sense that you can have the "actual amount of data saved" as one value, and "the space currently taken up on the disk" and "maximum allowed size" as seperate values. However for sparse bundles, they generally take up the same amount of disk as the data they contain (plus a small overhead). You min and cur values are the same – jksoegaard Apr 2 at 7:26
  • because the format automatically ensures that the amount of disk space taken up corresponds to the actual amount of data stored. The data is saved in small bands (default is 8.4 MB) unlike some other image format where you have to take up for example 100 GB on the disk to ensure you can later store that amount of data - no matter how much data you're actually storing right now. – jksoegaard Apr 2 at 7:28
0

I have possibly mixed up the concepts of min, cur and max.

I had assumed that cur represented the space currently used on the host volume, and not the current total logical space of the image. In the latter case a resize would still be necessary in order to increase available logical size, which in hindsight turns out to have had the desired effect.


To clarify the problem for future readers (including myself):

The root cause of my problem was that I've gotten the impression that sparsebundles will "expand" automatically, without getting a clear picture of what exactly will expand.

Since I was never particularly concerned with the fact that something that's sparse will potentially occupy less space than its advertised internal space — because that's only ever to my benefit — I had just assumed that mentions of how sparsebundles can expand automatically were referring to a separate concept, e.g. expanding its internal volume's space. That's something that would have been useful to know since it would have meant that I wouldn't have to worry about that the bundle's volume was filling up, as long as there's space left on the host volume.

In other words, while I could differentiate between the bundle's physical size on disk and its internal volume's size, I thought the latter had the capability to grow automatically. The hdiutil output did not help with that misconception.

Unsurprisingly (in hindsight) this is not how it works. The volume size is fixed until manual intervention, and knowing that its physical size "expands" automatically is only marginally useful, at least for my use case.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .