11

I am developing a program in go; it occasionally ends up allocating very large amounts memory (>>10G on a machine with 8G of physical memory), causing the system to become unresponsive. I want to limit the amount of memory the process can allocate. The usual way I would do this is:

ulimit -m 4000000 ; ./myprogram

...which should kill my program if it tries to use more than 4GB of memory.

On OS X El Capitan this appears to have no effect; even ulimit -m 1 (limiting all programs to just 1kB of memory!) is ineffective.

How can I set an upper bound on the memory available to a particular process?

  • When you say The usual way I would do this, what do you mean? More specifically, when do you usually do this? And when you do it, do you mean within Mac OS X El Capitan or in another environment? Can you give an example of when you've used it successfully? Basically, I'm just trying to clarify whether this process works for you normally, but just isn't working for this particular program? Or is it not working for you at all but you think it should? – Monomeeth Mar 28 '17 at 3:30
  • 1
    By "the usual way I would do this", I mean the way I would do this on other flavours of unix. I note that I have just discovered that ulimit -m no longer works on Linux (> 2.4.30), though ulimit -v does still work as expected. (Like ulimit -m, ulimit -v also appears to have no effect on OS X.) – cpcallen Mar 29 '17 at 12:16
  • That sounds like you've got a memory leak in the program you're writing and need to do better garbage collection. Have you read up on memory management in go? – Todd Dabney Apr 16 '17 at 18:04
  • 1
    No, not a memory leak—just a graph search of a potentially very large state space. I could add code to abort the search if the various internal structures get too large, but I was expecting to be able to do the same thing (prevent machine from getting wedged by large inputs) with a shell one-liner. – cpcallen Apr 17 '17 at 1:49
1

There are two approaches to constraining your memory usage: Ex post facto, and preemptive. That is to say, you can try to kill your program after it has gotten too large, or you can program it not to get too large in the first place.

If you insist on the ex post facto approach, you can use the following Bash script. This script first finds the amount of memory (as defined by "resident set size") the process with processid pid is using, filters out all non numerical data using grep, and saves the amount as variable n. The script then checks if n is greater than your specified x. If it is, the process with processid pid is killed.

Please note:

  1. You must replace <pid> with the process id of your program.
  2. You must replace <x> with the rss = "resident set size" (ie real memory size) you don't want the program to exceed.

n=$(ps -<pid> -o rss | grep '[0-9]') if [ $n -gt <x> ]; then kill -9 <pid>; fi

If you want this to run every y seconds just inclose it in a loop, and tell it to wait for y seconds after every iteration. You could also write a similar command using top. Your starting point would be top -l 1|grep "<pid>"|awk '{print $10}'.

@kenorb's answer helped me with my script


While I believe that answers the question, in the long run I believe it is better programming design to take a preemptive approach using manual memory allocation.

Firstly, are you sure the memory usage is really a problem? The Go documentation states:

The Go memory allocator reserves a large region of virtual memory as an arena for allocations. This virtual memory is local to the specific Go process; the reservation does not deprive other processes of memory.

If you still think you have a problem, then I encourage you to manually manage your memory as is done in the C programming language. Since go is written in C, I suspected there would be ways to get into the C memory management/allocations, and indeed there are. See this github repository which,

allows you to do manual memory management via the standard C allocator for your system. It is a thin wrapper on top of malloc, calloc and free from . See man malloc for details on these functions for your system. This library uses cgo.

The use case is given as:

Why would you want this?

When a program is causing memory pressure or the system is running out of memory it can be helpful to manually control memory allocations and deallocations. Go can help you control allocations but it is not possible to explicitly deallocate unneeded data.

This seems like a better long term solution.

If you want to learn more about C (including memory management), The C Programming language is the standard reference.

  • I am sure that memory usage is really a problem. When the program unexpectedly grew to be >2x physical memory size the machine became almost entirely non-responsive due to page thrashing. It took about an hour between when I hit ^C and when macOS resumed responding to mouse clicks; in the mean time the only evidence that it was not frozen was the quiet noise of the hard drive rapidly ticking about. – cpcallen Jun 5 '17 at 15:30
  • This is probably a reasonable solution in practice, but unlike ulimit on (non-Darwin) UNIX OSes it depends on being able to allocate enough memory in a timely fashion to successfully execute the kill command. I'd really rather have process size limits enforced by the kernel. – cpcallen Jun 5 '17 at 15:32
  • @cpcallen doing some searching it seems the "-m parameter to ulimit has no effect on Linux systems with kernel versions more recent than 2.4.30." It seems OSX has also picked up on this change. Try the -v option to limit the address space – Evan Rosica Jun 5 '17 at 23:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .