Matplotlib and multiprocessing RuntimeError

matplotlib, multiprocessing, python, tk-toolkit

Solution

The error messages reference `Tkinter`. So it looks like you are using the TkAgg backend. The code that follows is TkAgg/Tkinter-specific. In particular the call

win.after(100, animate)

makes use of the Tkinter-specific `after` method. There is an analogous call for GtkAgg/PyGtk, and similarly for other backends. But I just want to stress that what follows is TkAgg/Tkinter-specific.

Tkinter is intended to be run in a single thread. That is, all Tkinter GUI calls should originate from a single thread (usually, be not necessarily, the main thread).

The Pool's `apply_async` callback method runs in a separate (`_handle_results`) thread in the main process. Since `imshow()` is called from the Pool's `_handle_results` thread and `show()` is called in the main thread, `Tkinter` complains

RuntimeError: main thread is not in main loop

I don't see a way to use the `apply_async` callback in this situation.

Instead, what we can do is arrange for `do_work` to put `out_image` in a `multiprocessing.Queue()` (which I call `out_queue` in the code below). We'll then have the main process's main thread poll this Queue for items and display them as they come out of the Queue. This polling is done in the `animate` function, below.

`plt.ion()` is meant for interactive sessions only. Although it is sometimes possible to write little scripts which sort-of seem to work with `plt.ion()`, you'll get better results and cleaner GUIs if you resist using `plt.ion()` in scripts and instead write code that respects the GUI framework's event loop.

Although it is probably possible to fix your script and use `plt.ion()`, since this is not the recommended way of writing matplotlib scripts, let's see if we can avoid doing that.

`plt.show()` tells Tkinter to run its event loop. Notice that once this call is made, the GUI window is drawn, you can click buttons, zoom in and out, etc.

Somehow we need to inject a function into this event loop, to be run periodically by the event loop, and cooperatively with all the other GUI events that might be occurring. We want this function to check if any of our worker subprocesses have output for us, and if so, to update the imshow image.

With TkAgg/Tkinter, the way to inject such a function is

win = fig.canvas.manager.window
win.after(100, animate)

This will tell Tkinter to run the function `animate` (once) after (about) 100ms have elapsed. Since we want the function `animate` to run periodically, we just stick another

win.after(100, animate)

call at the end of `animate`.

import matplotlib as mpl
mpl.use('TkAgg')
import matplotlib.pyplot as plt
import numpy as np
import multiprocessing as mp
import logging
import Queue
logger = mp.log_to_stderr(logging.INFO)

# Only does something to in_image, doesn't access anything else
def do_work(in_image):
    logger.info('Processing in_image')
    for x in xrange(100000):
        out_image = in_image[::-1, ::-1]
    out_queue.put(out_image)

# Update the output image and display if needed
out_all = np.zeros((256, 256))


def pool_initializer(out_queue_):
    # Setup out_queue as a global variable *in the worker subprocesses*
    global out_queue
    out_queue = out_queue_


def animate():
    global out_all
    try:
        out_image = out_queue.get_nowait()
    except Queue.Empty:
        pass
    else:
        logger.info("Updating")
        out_all += out_image
        im.set_data(out_all)
        fig.canvas.draw()  # redraw the canvas
    win.after(100, animate)

if __name__ == '__main__':
    out_queue = mp.Queue()
    logger.info("Starting pool")
    pool = mp.Pool(initializer=pool_initializer, initargs=(out_queue, ))
    work = [np.random.random((256, 256)) for f in range(20)]
    for o in work:
        pool.apply_async(do_work, [o])
    pool.close()

    fig, ax = plt.subplots()
    win = fig.canvas.manager.window
    # Output image
    im = plt.imshow(out_all, vmin=0, vmax=1)

    # Register a function to be run once
    win.after(100, animate)
    plt.show()
    logger.info("Done")

Problem

Original source