How can I increase the maximum allowed swap space?

Question

When I try to do a calculation that needs to allocate and use 128 GB of memory (it is a command line program written in C), the kernel kills my process with extreme prejudice. This console log entry is an example of one instance:

9/25/15 7:08:40.000 PM kernel[0]: low swap: killing pid 6202 (huffgrp)

The calculation works fine and in a reasonable amount of time when it allocates and uses 64 GB of memory. My machine has 32 GB of RAM and beaucoup space on the hard drive. I also tried this on another machine with 8 GB of RAM, on which the 64 GB calculation runs fine as well, taking longer of course, but the 128 GB calculation gets killed by the kernel in the same way. On both I am running Yosemite 10.10.5.

By the way, malloc() never returns an error, no matter how much space I ask for. The kernel will only kill the process once too much of that memory is actually being used by the process, resulting in lots of swapping to the hard drive.

So there appears to be a secret swap space limit somewhere between 64 GB and 128 GB.

My question is: how do I reconfigure the kernel to permit more swap space? I found a promising looking file, /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist, but I don't see the secret number in there. The man page for dynamic_pager says that all it does is set the name and location of the swap files. There is an older version of the same man page that documents a -S option to set the size of the swapfiles created. I tried that, requesting 160 GB swapfiles, but it had no effect. The swapfiles were still 1 GB each, and the process was still killed by the kernel.

Answer 1

Well, it's been 7 years after this question was asked, but I just spent quite some time on this question, and at least as of Jun 2021 (macOS Monterey, commit 2ff845c2), the swap size limit is hardcoded 100 GB (100 files, 1 GB each). The behavior I observe on Ventura seems identical.

See vm_compressor_backing_store.c line 150:

#define VM_MAX_SWAP_FILE_NUM            100

and vm_compressor_backing_store.h line 50:

#define MAX_SWAP_FILE_SIZE              (1 * 1024 * 1024 * 1024ULL)

Lines are copied for clarity, but these line numbers can't change because the links point to the specific commit hash.

Answer 2

While I could not find any more recent documentation, Singh's "Mac OS X Internals: A Systems Approach" mentions two obscure limits in /sbin/dynamic_pager (p. 919 f.):

• The absolute minimum and maximum swap file sizes are 64MB and 1GB, respectively.

• The maximum swap file size must not be greater than 12.5% of the free space available on the volume that contains the swap file. Moreover, the maximum swap file size must not be greater than the amount of physical memory on the system.

• At most, eight swap files can be created.

[...]

While the book is a bit dated and the numbers are likely to have changed, the 12.5% rule sounds very suspicious to me.

Singh implies that those are hard-coded limits in the binary and thus the property file you found is likely not to be of use. Also note that the swap system creates several, but only up to eight, distinct swap files.

/sbin/dynamic_pager is documented, as you noted, as a very rudimentary man page, and there is a conflicting statement in the current developer documentation on Virtual Memory, stating that

To give processes access to their entire 4 gigabyte or 18 exabyte address space, OS X uses the hard disk to hold data that is not currently in use. As memory gets full, sections of memory that are not being used are written to disk to make room for data that is needed now. The portion of the disk that stores the unused data is known as the backing store because it provides the backup storage for main memory.

[...]

Note: Unlike most UNIX-based operating systems, OS X does not use a preallocated disk partition for the backing store. Instead, it uses all of the available space on the machine’s boot partition.

(emphasis mine)

Hence, at around this point, I'd go for some pragmatic workaround. Perhaps try increasing your disk space even further to check for a correlation in the obtainable swap size, thus checking the 12.5% limitation mentioned above.