Catching signals sent by runit (sv stop <id>)
python, runit
Solution
Okay, so after some extensive testing I figured it out.
Runit will send the kill signal to the `run` script, which does not propogate it by default. What you need to make sure is that you call `exec python yourscript.py` in the end. Similarly, if your `run` script calls another shell script (ie. one that activates your virtualenv or similar), it must do so with `exec` as well.
Samples:
`run`:
#!/bin/sh
umask 002
2>&1
exec chpst -uanalytics cliscript router
`cliscript`:
#!/bin/sh
# Resolve script path, assuming that the script resides in $(ABSPATH)/bin
SCRIPTPATH="$0"
if [ -h "$SCRIPTPATH" ]; then
SCRIPTPATH=$(readlink -e "$0")
fi
ABSPATH=$(dirname "$(cd "$(dirname "$SCRIPTPATH")"; pwd -L)")
# Load the virtual environment
source "$ABSPATH/venv/bin/activate"
# Set up environment
export PYTHONUNBUFFERED=1
exec python "$ABSPATH/bin/processing-cli.py" $@
Take note of the `exec` being called when we "pass" control to the next script or python itself.
Problem
I have a python program that is running a number of worker processes. Because this has to be handled properly to avoid orphaned processes, I have implemented a signal handler to shutdown all worker processes. The program starts more or less like this: - Start process pool (starts X number of workers) - Register signal handlers (`signal.signal(signal.SIGTERM, my_signal_handler)`). I also add another signal handler for `SIGINT` with the same handler. - Start seperate thread polling backend (database) and add the tasks to process pool. - On main thread, poll process pool for results (there is a result `multiprocessing.Queue` that the individual workers add the results to). The idea is that the two seperate threads started in 3 and 4 keep the tasks running through the machinery. If I start this manually and call `kill -15 <pid>` or `kill -2 <pid>` it correctly shuts everything down, waits for processes to `join()`. Reading from the documentation, runit sends a `TERM` to the process, followed by `CONT`. However, running this under runit, it simply shows the standard `ok: down: <my_program>: 1s, normally up`, but the process is still running in the background (even the main process, it is UNTOUCHED). If I then afterwards go out and manually kill the process, I can see in the log file that it shuts down correctly. What am I doing wrong? It seems that runit ONLY kills the 3-line shell script I created to activate the virtualenv, but leaves the actual python process behind. Even if I run the "run" script directly, I can either run `kill` or Ctrl+C (same as `SIGINT`) and it shuts down correctly.