How to find out what/who exactly killed a python process?

Question

I ran a piece of python code like this python -m example.main in the terminal.

It ran for like 10 mins and then I see zsh: killed python -m example.main

I already checked kernel log like this: dmesg | greg 1101 where 1101 was the python PID process. And it showed nothing. So I believe it's not related to OOM.

I believe "zsh: killed..." means the process was killed by something with SIGKILL signal. Unfortunately, there's no way to catch SIGKILL signal inside python program to tell who issues the signal.

Is there a way to tell exactly who/what process issues the SIGKILL to my python program? How do I monitor for this?

Answer 1

You can use a system tracing tool, such as dtrace, to observe the kill syscall being invoked in the kernel, which is how POSIX signals are sent. You are primarily interested in SIGKILL, which is signal number 9.

Here is a dtrace script that will monitor all signals sent from any process to any other process on your system. To use it:

1. Save this script as a file with a .d extension.
2. Give the script execute privileges (chmod u+x SCRIPTNAME.d).
3. Run the script as a superuser (sudo ./SCRIPTNAME.d).
4. Start your Python program.
5. Find the PID of your Python program (e.g., via ps or top) and make note of it.
6. When you see that your Python program has been killed, find its PID in the signal log, confirming that you see signal number 9, and you'll find the name and PID of the process that sent it.

#!/usr/sbin/dtrace -qs

syscall::kill:entry
{
    self->target = arg0;
    self->signal = arg1;
}

syscall::kill:return /self->signal != 0 && self->target > 0/
{
    printf("%s (%d) --> % 6d:  signal %d\n", execname, pid, self->target, self->signal);

    self->target = 0;
    self->signal = 0;
}

dtrace:::BEGIN
{
    printf("SOURCE (PID) --> TARGET:  SIGNAL\n");
    printf("================================\n");
}