Python multiprocessing Queue put() behavior
multiprocessing, python, python-multiprocessing, python-multithreading, queue
Solution
Actually, this is thought to be a feature, not a problem. The queue immediately returns so your process continues while serialization happens and to avoid what is known as "queue contention".
The two options I suggest you have:
Are you absolutely sure you need mutable dictionaries in the first place? Instead of making defensive copies of your data, which you correctly seem to dislike, why not just create a new dictionary instead of using `dict.clear()` and let the garbage collector worry about old dictionaries?
Pickle the data yourself; That is: `a_queue.put(pickle.dumps(data))` and `pickle.loads(a_queue.get())`. Now, if you do `data.clear()` just after a `put`, the data has already been serialized "by you".
From a parallel programming point of view the first approach (treat your data as if it were immutable) is the more viable and clean thing to do on the long term, but I am not sure if or why you must clear your dictionaries.
Problem
I'm doing something very simple using multiprocessing: ``` data = {'a': 1} queue.put(data, True) data.clear() ``` When I use the queue on another process (using `get()` method), I get an empty dictionary. If I remove `data.clear()` I get the keys as expected. Is there any way to wait for the `put()` to have finished the serialization ?