Why does this python queue code process items multiple times?
multithreading, python, queue
Solution
The Queue package is not process aware, it only works for threads. The following happens in your example:
- Create Queue and fill with numbers
- Fork 4 processes. This copies the memory content into each subprocess, including the filled Queue
- Each process empties its copy of the queue
You have to use the Queue class provided by multiprocessing.
Problem
The following is a testcase I created. Why does every process print the number 1 to 5 and are the numbers not divided over the processes? code: ``` #!/usr/bin/python from subprocess import * from Queue import Queue from Queue import Empty import multiprocessing from multiprocessing import Process def main(): r = Runner() r.run() class Runner(object): processes = [] def run(self): q = Queue() for t in range(1,6): q.put(t) for pi in range(1,4): p = Process(target=self.runFromQueue, args=(q,)) p.start() self.processes.append(p) for p in self.processes: p.join() print "Finished!" def runFromQueue(self, q): try: while True: number = q.get_nowait() print str(number) q.task_done() except Empty: pass if __name__ == "__main__": main() ``` Ouput: ``` $ ./test_threads.py 1 2 3 4 5 1 1 2 3 2 4 3 5 4 5 Finished! ``` Expected ouput: ``` $ ./test_threads.py 1 2 3 4 5 Finished! ```