4

I've been building a Neural Network that analyzes large amounts of data (40G), and my iMac kills the process after it's been running for about a day.

In the past, on Linux, I've created a large swap file to get around memory limitations.

I see:

$ python processor.py
[...maybe some std out messages, specific to what I'm doing...]
Killed.

I've come to know this as the "you've used up too much memory, good bye" message. Again, I've been able to solve it using a large swap file on linux. How can I increase the swap limit on my mac so my processes that use large amounts of memory don't get killed?

Not sure how I could get more information about why it got killed.

Improve this question
3
  • 2
    Could you edit in exactly the system messages that are logged when the process is killed? What is the memory status at that point in time (maybe 5 minutes before and 5 minutes after termination?) My guess is you have another problem as swap can and will grow to take up all space on the boot volume, but usually it's the program's fault to have that much RAM swapped out and never back in again.
    – bmike ♦
    Commented Jan 15, 2015 at 23:25
  • 1
    Added some edits above.
    – InBetween
    Commented Jan 16, 2015 at 0:08
  • Nice edits. Since you're running it from a shell, also consider looking at ulimit -a I don't have a test case where I can cause the system to choke on 10.10.x Also, watch the memory pressure
    – bmike ♦
    Commented Jan 16, 2015 at 1:02

2 Answers 2

Reset to default
5

Here are some ways to check for issues but I fear they may not be a complete or even the correct solution without more peeking or poking:

In another shell after you start your python process (or if you screen / tmux it):

  1. df /
  2. top -l 1 -S | head -12
  3. vm_stat and vm_stat 600
  4. sudo du -sm /var/vm/*

Once you have a good baseline, you can watch things over time to see how the neural net is behaving each hour for a while. If you think things are about stop, you can run sysdiagnose python (or use the process # if you have more than one python process running). Also, if you don't want to wait the day for things to bulk up, you can inflict memory_pressure on the system before or after starting the neural net in python. See this answer for how to monitor the Activity Monitor when you run this process:

Improve this answer
4
  • Awesome, thanks for this. I'll play around with it and see what I find.
    – InBetween
    Commented Jan 16, 2015 at 18:15
  • 1
    I am pretty sure that you mean df / where you wrote df \
    – Pascal Cuoq
    Commented Jan 16, 2015 at 22:11
  • @PascalCuoq You're right. Feel free to edit posts where this is the case in the future :-) Sometimes, the system is cranky about small edits though.
    – bmike ♦
    Commented Jan 16, 2015 at 22:40
  • @bmike Yes, on SE sites where one is only starting with 100 reputation, one needs to find at least 6 characters to change for the edit to be considered significant and enter review. I have done it but it can be hair-pulling: crypto.stackexchange.com/posts/18651/revisions
    – Pascal Cuoq
    Commented Jan 16, 2015 at 22:57
0

It's SIGKILLed. It's hard to tell it's OOM on macOS since when you tail -f /var/log/system.log

 Jan 14 10:46:39 ... com.apple.xpc.launchd[1] (com.apple.mdworker.shared.10000000-0000-0000-0000-000000000000[41845]):           Service exited due to SIGKILL | sent by mds[72]

The error message doesn't help.

Then I tried Instrument shipped with macOS,

enter image description here

And attach instrument to your Python process. You can see the memory goes unbounded until it's killed.

enter image description here

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .