Maximum size for multiprocessing.Queue item?

multiprocessing, multithreading, python, queue

Solution

Seems the underlying pipe is full, so the feeder thread blocks on the write to the pipe (actually when trying to acquire the lock protecting the pipe from concurrent access).

Check this issue http://bugs.python.org/issue8237

Problem

I'm working on a fairly large project in Python that requires one of the compute-intensive background tasks to be offloaded to another core, so that the main service isn't slowed down. I've come across some apparently strange behaviour when using `multiprocessing.Queue` to communicate results from the worker process. Using the same queue for both a `threading.Thread` and a `multiprocessing.Process` for comparison purposes, the thread works just fine but the process fails to join after putting a large item in the queue. Observe: ``` import threading import multiprocessing class WorkerThread(threading.Thread): def __init__(self, queue, size): threading.Thread.__init__(self) self.queue = queue self.size = size def run(self): self.queue.put(range(size)) class WorkerProcess(multiprocessing.Process): def __init__(self, queue, size): multiprocessing.Process.__init__(self) self.queue = queue self.size = size def run(self): self.queue.put(range(size)) if __name__ == "__main__": size = 100000 queue = multiprocessing.Queue() worker_t = WorkerThread(queue, size) worker_p = WorkerProcess(queue, size) worker_t.start() worker_t.join() print 'thread results length:', len(queue.get()) worker_p.start() worker_p.join() print 'process results length:', len(queue.get()) ``` I've seen that this works fine for `size = 10000`, but hangs at `worker_p.join()` for `size = 100000`. Is there some inherent size limit to what `multiprocessing.Process` instances can put in a `multiprocessing.Queue`? Or am I making some obvious, fundamental mistake here? For reference, I am using Python 2.6.5 on Ubuntu 10.04.

Original source

Related problems