Multiprocessing a for loop?

multiprocessing, python

Solution

You can simply use `multiprocessing.Pool`:

from multiprocessing import Pool

def process_image(name):
    sci=fits.open('{}.fits'.format(name))
    <process>

if __name__ == '__main__':
    pool = Pool()                         # Create a multiprocessing Pool
    pool.map(process_image, data_inputs)  # process data_inputs iterable with pool

Problem

I have an array (called `data_inputs`) containing the names of hundreds of astronomy images files. These images are then manipulated. My code works and takes a few seconds to process each image. However, it can only do one image at a time because I'm running the array through a `for` loop: ``` for name in data_inputs: sci=fits.open(name+'.fits') #image is manipulated ``` There is no reason why I have to modify an image before any other, so is it possible to utilise all 4 cores on my machine with each core running through the for loop on a different image? I've read about the `multiprocessing` module but I'm unsure how to implement it in my case. I'm keen to get `multiprocessing` to work because eventually I'll have to run this on 10,000+ images.

Original source