The right way to limit maximum number of threads running at once?
multithreading, python, python-multithreading
Solution
It sounds like you want to implement the producer/consumer pattern with eight workers. Python has a `Queue` class for this purpose, and it is thread-safe.
Each worker should call `get()` on the queue to retrieve a task. This call will block if no tasks are available, causing the worker to go idle until one becomes available. Then the worker should execute the task and finally call `task_done()` on the queue.
You would put tasks in the queue by calling `put()` on the queue.
From the main thread, you can call `join()` on the queue to wait until all pending tasks have been completed.
This approach has the benefit that you are not creating and destroying threads, which is expensive. The worker threads will run continuously, but will be asleep when no tasks are in the queue, using zero CPU time.
(The linked documentation page has an example of this very pattern.)
Problem
I'd like to create a program that runs multiple light threads, but limits itself to a constant, predefined number of concurrent running tasks, like this (but with no risk of race condition): ``` import threading def f(arg): global running running += 1 print("Spawned a thread. running=%s, arg=%s" % (running, arg)) for i in range(100000): pass running -= 1 print("Done") running = 0 while True: if running < 8: arg = get_task() threading.Thread(target=f, args=[arg]).start() ``` What's the safest/fastest way to implement this?
Related problems
- Catch and print full Python exception traceback without halting/exiting the program
- How do I limit the number of active threads in python?
- How to make worker threads quit after work is finished in a multithreaded producer-consumer pattern?
- How to wait until only the first thread is finished in Python
- How to detect exceptions in concurrent.futures in Python3?