Python multiprocessing's Pool process limit
cpu-cores, multiprocessing, python
Solution
You can ask for as many processes as you like. Any limit that may exist will be imposed by your operating system, not by `multiprocessing`. For example,
p = multiprocessing.Pool(1000000)
is likely to suffer an ugly death on any machine. I'm trying it on my box as I type this, and the OS is grinding my disk to dust swapping out RAM madly - finally killed it after it had created about 3000 processes ;-)
As to how many will run "at one time", Python has no say in that. It depends on:
- How many your hardware is capable of running simultaneously; and,
- How your operating system decides to give hardware resources to all the processes on your machine currently running.
For CPU-bound tasks, it doesn't make sense to create more `Pool` processes than you have cores to run them on. If you're trying to use your machine for other things too, then you should create fewer processes than cores.
For I/O-bound tasks, it may make sense to create a quite a few more `Pool` processes than cores, since the processes will probably spend most their time blocked (waiting for I/O to complete).
Problem
In using the Pool object from the multiprocessing module, is the number of processes limited by the number of CPU cores? E.g. if I have 4 cores, even if I create a Pool with 8 processes, only 4 will be running at one time?