When I try to do a calculation that needs to allocate and use 128 GB of memory (it is a command line program written in C), the kernel kills my process with extreme prejudice. This console log entry is an example of one instance:
9/25/15 7:08:40.000 PM kernel: low swap: killing pid 6202 (huffgrp)
The calculation works fine and in a reasonable amount of time when it allocates and uses 64 GB of memory. My machine has 32 GB of RAM and beaucoup space on the hard drive. I also tried this on another machine with 8 GB of RAM, on which the 64 GB calculation runs fine as well, taking longer of course, but the 128 GB calculation gets killed by the kernel in the same way. On both I am running Yosemite 10.10.5.
By the way,
malloc() never returns an error, no matter how much space I ask for. The kernel will only kill the process once too much of that memory is actually being used by the process, resulting in lots of swapping to the hard drive.
So there appears to be a secret swap space limit somewhere between 64 GB and 128 GB.
My question is: how do I reconfigure the kernel to permit more swap space? I found a promising looking file,
/System/Library/LaunchDaemons/com.apple.dynamic_pager.plist, but I don't see the secret number in there. The man page for
dynamic_pager says that all it does is set the name and location of the swap files. There is an older version of the same man page that documents a
-S option to set the size of the swapfiles created. I tried that, requesting 160 GB swapfiles, but it had no effect. The swapfiles were still 1 GB each, and the process was still killed by the kernel.