Forcing `kernel_task` to swap memory?

Question

I have had a week-long compression process run on an MacBook. When it hit 95%, the speed suddenly dropped to just 5% CPU usage. Tracing the problem, I found that kernel_task is taking up 6.87GB, leaving roughly 1GB to the other processes. This leads to thrashing on the compression process. Tracing the issue further, I found that a bash process is using nearly 1TB (!) of kernel memory. This is slightly more than the free space on my hard drive! No clue what it's doing, no other process seems to have this as its parent.

I closed all bash windows, but the process stayed. Ran kill on it, no difference. I finally killed it using kill -9.

In Why does leaked memory appear malloced to kernel_task, and why can't OS X therefore garbage collect it, killing Preview resolved this issue, but in my case, the memory usage of kernel_task remained. Maybe due to how it was killed. Tried running sync and purge, no difference. Garbage collect does not apply to kernel_task. Any other things I could do to reclaim this memory? Theoretically, if I could get the inactive memory at least moved to swap, then the compression process would have enough to quickly complete what it's doing.

No kernel extensions seem to be using an abnormal amount of memory, no third-party extensions are installed.

Answer 1

You're unlikely to make any headway with this. All of the levers you might pull to change the system require a reboot save for one.

If you can pause the compression task and are using bash/ksh/zsh, you could disown the process so you are free to exit the graphical session, end the terminal process that spawned your shell. All memory by bash and everything else you started when you logged in will be freed. No garbage collection will happen, but that's the best you can do once you start the compression.

• https://stackoverflow.com/questions/625409/how-do-i-put-an-already-running-process-under-nohup

1. Ctrl+Z to stop (pause) the program and get back to the shell.
2. bg to run it in the background.
3. disown -h [job-spec] where [job-spec] is the job number (like %1 for the first running job; find about your number with the jobs command) so that the job isn't killed when the terminal closes.

In the future you can start these very long running tasks with tmux or screen or nohup so you can then log out of the graphical session and leave 100% of the resources to that process and it's children. Fixing memory leaks isn't really amenable to hot patching as you have to change the things you do after restart and/or patch the system and apps leaking memory.

---

Next run, I would ssh in to the Mac from another device (phone, iPad, other computer) and run your compression under screen and watch for leaks and other issues. Assuming it's clean, you could then log in graphically and try to minimize leaks if you needed to use the hardware for other tasks and then log out as soon as you're done.

In my experience, disabling swap or other tuning won't help get you around bad coding and leaks, but please report if you find a fix. You've got so much more detail on what you're doing than us, perhaps there's something you can do.

Answer 2

The answer by bmike is simply great, and lists things that you can do with a general process. I would like also to add something specific to this process, namely compression.

The idea is to end the compression where it is, so that after logging out and back in, you can continue the process to add more files.

One thing NOT to try is, to move away the files that have not been compressed. In my case, I've left the directory being compressed, but moved files from all other directories away, hoping it would see the empty directories, compress them, and finish. But apparently the list of actual files was compiled when the compression started. So after finishing the current directory, the system threw a File not found error for all the remaining files, and after finishing the entire list, it exited with a System error: E_FAIL. In other words, no archive file was left behind, and the entire week of compression was lost.

One way to end the compression where it is is, to create a parallel directory on the same drive, and create an empty file for each of the remaining files, via touch. Then, to swap the two directories in a single command. When the swap occurs, the file that started to compress has been opened, so it will finish even though it is now in a different directory. This way, no File not found error messages will be generated during the few milliseconds while the swap is happening. And when this file is finished, then the remaining files it will read will be empty, so they will compress almost instantaneously, thereby allowing the compression process to finish creating an archive file anyways. After logging out and back in, this archive can be simply updated with the correct files.