How is Python blocking signals while os.system("sleep...")?

linux, python, python-2.7, signals

Solution

Maybe the magic is in PY_BEGIN_ALLOW_THREADS?

The magic is mostly in `system` itself. `system` cannot return EINTR, so the libc implementation goes to pains to resume its `wait`'ing on the child process. That means in your use of `os.system`, control never returns to python until the underlying `system` completes, and thus the python signal handling mechanics aren't invoked timely.

`subprocess.call`, however, is essentially doing this:

# Compare subprocess.py:Popen/_eintr_retry_call(os.waitpid, self.pid, 0)
while True:
  try:
    return os.waitpid(the_child_pid, 0)
  except OSError, e:
    if e.errno == errno.EINTR:  # signal.signal() handler already invoked
      continue
    raise

Here control does return to python when the underlying `wait` is interrupted. The OSError/EINTR prompts python to see if any signals were tripped and, if so, to invoke the user-supplied code block associated with that signal. (And that's how the interpreter adapts the system's signal semantics: set a flag, and check it between "atomic" python operations, invoking the user's code if appropriate.)

Problem

When I run this Python script with `os.system` on Ubuntu 12.04: ``` import os, signal signal.signal(signal.SIGABRT, lambda *args: os.write(2, 'HANDLER\n')) print 'status=%r' % os.system('sleep 5') ``` , and then I send SIGABRT to the script process many times within 5 seconds, I get the following output: ``` status=0 HANDLER ``` This indicates that the signal delivery was blocked until `sleep 5` exited, and then only a single signal was delivered. However, with `subprocess.call`: ``` import os, signal, subprocess signal.signal(signal.SIGABRT, lambda *args: os.write(2, 'HANDLER\n')) print 'cstatus=%r' % subprocess.call('sleep 5', shell=True) ``` , all individual signals are delivered early: ``` HANDLER HANDLER HANDLER cstatus=0 ``` To distinguish the magic in glibc from the magic in Python, I rewrote the Python script in C, so `os.system` became system(3): ``` #include <errno.h> #include <signal.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <unistd.h> static void handler(int signum) { (void)signum; write(2, "HANDLER\n", 8); } int main(int argc, char **argv) { int got; struct sigaction sa; (void)argc; (void)argv; memset(&sa, 0, sizeof sa); sa.sa_handler = handler; if (0 != sigaction(SIGABRT, &sa, NULL)) return 3; got = system("sleep 5"); return !printf("system=0x%x\n", got); } ``` Signals got delivered early: ``` HANDLER HANDLER HANDLER system=0x0 ``` So I inferred that the magic is in Python 2.7, not in eglibc. But where is the magic? Based on the strace output and looking at the `posix_system` function in `Modules/posixmodule.c`, I couldn't figure out how Python blocks the signal until `os.system` returns. Relevant code from `Modules/posixmodule.c`: ``` static PyObject *posix_system(PyObject *self, PyObject *args) { char *command; long sts; if (!PyArg_ParseTuple(args, "s:system", &command)) return NULL; Py_BEGIN_ALLOW_THREADS sts = system(command); Py_END_ALLOW_THREADS return PyInt_FromLong(sts); } ``` Maybe the magic is in `Py_BEGIN_ALLOW_THREADS`? Do I understand correctly that it's impossible for my Python signal handler (set up by `signal.signal`) to execute until `os.system` returns? Is it because signal handlers are blocked (on the Python level, not on the OS level) until `Py_END_ALLOW_THREADS` returns? Here is the strace output of the Python code with `os.system`: http://pastebin.com/Wjn9KBye

Original source