How do I automatically kill processes that leak memory?

Question

I have problems with processes that leak memory. They cause my hard drive to be filled with swap files in /private/var/vm.

I would like leaking processes to be killed at sight by the OS. I’m not interested in a dialog that appears after 20 minutes, suggests applications to kill, and doesn’t even show the one that is leaking.

I have tried setting the rss and data limits in /etc/launchd.conf but it didn’t seem to have an effect.

This is my /etc/launchd.conf:

limit data 8589934592 8589934592
limit rss 8589934592 8589934592

Here is the launchctl limit output:

% launchctl limit
    cpu         unlimited      unlimited
    filesize    unlimited      unlimited
    data        8589934592     8589934592
    stack       8388608        67104768
    core        0              unlimited
    rss         8589934592     8589934592
    memlock     unlimited      unlimited
    maxproc     709            1064
    maxfiles    256            unlimited

This is my .zshrc:

ulimit -t 600
ulimit -d 512000
ulimit -v 1024000

And the ulimit -a output (in ZSH):

% ulimit -a
    -t: cpu time (seconds)         600
    -f: file size (blocks)         unlimited
    -d: data seg size (kbytes)     512000
    -s: stack size (kbytes)        8192
    -c: core file size (blocks)    0
    -v: address space (kb)         1000
    -l: locked-in-memory size (kb) unlimited
    -u: processes                  709
    -n: file descriptors           256

Yet here is what top tells me about the process:

PID  COMMAND      %CPU  TIME     #TH  #WQ  #POR #MRE RPRVT  RSHRD  RSIZE  VPRVT  VSIZE  PGRP PPID
886  process      30.8  01:16.40 1/1  0    17   332  2259M+ 184K   2072M- 19G+   38G    882  885

It just seems that none of the documented memory limitation methods actually work. Is there an additional mechanism that I may have missed?

Answer 1

I would create a simple script which would filter process any process with a resident memory size (or perhaps total vm size, so including any paged out pages) bigger than a threshold that I define (depending on the amount of process, total memory available and perhaps also CPU availability). One can use a bit of bash script with either top or ps to dig out the list of process and memory size.

From this filtered list, I would use the leaks command (see man 1 leaks) per process PID. If the total amount of leaked memory reported by the command is higher than yet another threshold, I would then kill and respawn it.

NOTE: You should take care not to kill any OS/System processes without knowing what you do. To avoid this situation, you should perhaps filter out the list using a "white list" approach.

Answer 2

ulimit on most platforms does not work as expected.

If this is not a desktop app, run the offender with a proper process supervisor such as https://github.com/arya/bluepill

If this is a desktop app, contact the app developers. Feedback is necessary and important.

The UX on the OOM killer GUI for Mac is horrible. It should be biggest sorted descending with a proportional bar graph for each processes' ram usage. Furthermore, it should automatically SIGCONT all paused processes when resolved.

Answer 3

If you need this across reboots you should use only launchctl limit command.

If you need to limit application's memory you should also limit stack segment.