Get number of busy CPUs in Python
multiprocessing, python
Solution
There are a number of complications...
- you can't determine which CPUs are busy
Processes (and threads) are scheduled by the Linux kernel on any CPU. Even determining the "current CPU" is awkward -- see How can I see which CPU core a thread is running in?
- `multiprocessing.Pool` is designed to start up N workers, which run "forever." Each accepts a task from a queue, does some work, then outputs data. A `Pool` doesn't change size.
Two suggestions:
- the uptime command outputs something like this:
`19:05:07 up 4 days, 20:43, 3 users, load average:`0.99`, 1.01, 0.82`
The last three numbers are the "load average" over the last minute, five minutes, and 15 minutes. Consider using the first number to load-balance your application.
- consider having your application do `time.sleep(factor)` after completing each piece of work.
Thus you can increase the factor when the system is busy (high load average), and make the delay shorter when the system is more idle (low load; ie surfing). Pool stays same size.
Problem
I am writing a `multiprocessing` routine to run on a server with plenty of CPUs. However, the server has multiple users and its usage may vary. So I would like to adapt the number of processors being used according to the current load. - Is there a way to estimate the amount of CPUs currently busy in Python? I only found `multiprocessing.cpu_count()` - Bonus question: Is it possible to change `multiprocessing.Pool(processes=no_cpus)` during activity, in case the load on the server has changed after a while?