Deadlock with logging multiprocess/multithread python script
logging, multiprocess, multithreading, python
Solution
This is probably bug 6721.
The problem is common in any situation where you have locks, threads and forks. If thread 1 had a lock while thread 2 calls fork, in the forked process, there will only be thread 2 and the lock will be held forever. In your case, that is `logging.StreamHandler.lock`.
A fix can be found here (permalink) for the `logging` module. Note that you need to take care of any other locks, too.
Problem
I am facing the problem with collecting logs from the following script. Once I set up the `SLEEP_TIME` to too "small" value, the LoggingThread threads somehow block the logging module. The script freeze on logging request in the `action` function. If the `SLEEP_TIME` is about 0.1 the script collect all log messages as I expect. I tried to follow this answer but it does not solve my problem. ``` import multiprocessing import threading import logging import time SLEEP_TIME = 0.000001 logger = logging.getLogger() ch = logging.StreamHandler() ch.setFormatter(logging.Formatter('%(asctime)s %(levelname)s %(funcName)s(): %(message)s')) ch.setLevel(logging.DEBUG) logger.setLevel(logging.DEBUG) logger.addHandler(ch) class LoggingThread(threading.Thread): def __init__(self): threading.Thread.__init__(self) def run(self): while True: logger.debug('LoggingThread: {}'.format(self)) time.sleep(SLEEP_TIME) def action(i): logger.debug('action: {}'.format(i)) def do_parallel_job(): processes = multiprocessing.cpu_count() pool = multiprocessing.Pool(processes=processes) for i in range(20): pool.apply_async(action, args=(i,)) pool.close() pool.join() if __name__ == '__main__': logger.debug('START') # # multithread part # for _ in range(10): lt = LoggingThread() lt.setDaemon(True) lt.start() # # multiprocess part # do_parallel_job() logger.debug('FINISH') ``` How to use logging module in multiprocess and multithread scripts?