0

I have a problem with a python script that is getting killed due to, what I believe, low available memory but I hope is a bug.

Description as follows: I have a python script that loads 12 large (1-1.5 Gb) files into memory. The reason for this is that each is a experiment file and I am doing cross-experiment analysis. I could do this without a problem on my current machine when it ran OSX High Sierra. But since I upgraded to Mojave the script fails.

The script fails during the loading of the files. A warning is issued (see image) which states that my disk space is critically low. I have previously been able to run the script with less than 50Gb free disk space. I have now cleared a lot of data and currently have 300+ Gb free space. A few seconds after the warning is issued the script is Terminated.

Right before the termination, if i follow the script process in Activity Monitor, the Virtual Memory Size is at ca 140 Gb, Memory Used is at 13.61 Gb, Cached Files at 2.0 Gb and Swap Used is at 11.43 Gb. In the Terminal there is no stack trace, it just says “Killed: 9”. So, I guess that this is not a issue with python but with the OS.

Warning message prior termination

I run: MacBook Pro (Retina, 15-inch, Mid 2015) 2.5 GHz Intel Core i7 16 Gb 1600 MHz DDR3 RAM

The python env is a anaconda virtual env conda 4.5.11 python 3.6.3

Since I dont get any stack trace and the only warning is the “critically low disk space” one, Im not sure how to attack this issue. Of course I can rewrite the script and to a lazy loading of partial data as needed, but that will take alot of time and root of this problem would be unsolved.

Any suggestions?

EDIT: After a night in my backpack it now runs. No idea why.

  • 1
    The problem is that that is a lot of VM. Yes in theory it s OK but the program would probably be significantly quicker to run and take less memory if it did not load all the data at once. I would ask what is the disk space when the crash happens. – Mark Jan 10 at 19:33
  • I agree with you that the script could have been designed in a better way. But (in the beginning of time when I wrote it) the plan was to use it once and then have the results and then be done with it. So I traded time writing the script against time executing it. Anywhoo, I followed disk usage and it went down to 298 Gb left on the HDD. And lo and behold, now it runs... ? I have not done anything. I tried it 10 times yesterday, with and without rebooting prior and always the same error. Now after a nights rest in my backpack it runs. Thank you for your answer anyway. – JMDE Jan 11 at 10:26

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.