Python os.fork OSError : [Errno 12] Cannot allocate memory (but memory not the issue)
linux, memory, out-of-memory, python
Solution
Hypothesis: if your VM is 32-bit, you may be running out of address space.
Not memory: address space. Let me explain: in Linux many things (IO, video card, memory-mapped files) use up address space without necessarily consuming corresponding amount of main memory.
Here's an explanation of related issues:
http://us.download.nvidia.com/XFree86/Linux-x86/331.89/README/knownissues.html
(look for "Kernel virtual address space exhaustion on the X86 platform" section, use `dmesg` to test if that's the situation)
`ENOMEM` error in result of `mmap` may very well mean situation of "not enough address space", not just "not enough memory", although I'm not sure how to diagnose this in CPython. If you have some big files `mmap`ed on your system by any process running on it, well..
Problem
I have similar problem to this one: Python subprocess.Popen "OSError: [Errno 12] Cannot allocate memory" I have a daemon process that runs OK for a few minutes and then fails to run shell programs via `popen2.Popen3()`. It spawns 20 threads. Memory does not appear to be the issue; this is the only program running on the machine, which has 2G of RAM, and it's using less than 400M. I've been logging ru_maxrss and this is only 50M (before and after OSError is raised). ``` ulimit -a: core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 15962 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 15962 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited ``` I've also been watching `free -m` and `ls /proc/$pid/fd | wc -l` while it is running, and neither of these seems to indicate resource exhaustion. Here's typical `free -m` while running: ``` total used free shared buffers cached Mem: 2003 374 1628 0 46 154 -/+ buffers/cache: 173 1830 Swap: 283 0 283 ``` ... and the fd count is around 90-100. The host is Ubuntu 12.04 (server jeos - minimal vm), Python 2.7.3, running on a VMWare host. So I'm wondering: what do I do next to diagnose why this is failing? Are there some more resource stats I can gather? Do I need to get down to the level of strace?