Perform a for-loop in parallel in Python 3.2
multiprocessing, parallel-processing, pickle, python, python-3.x
Solution
Joblib is designed specifically to wrap around multiprocessing for the purposes of simple parallel looping. I suggest using that instead of grappling with multiprocessing directly.
The simple case looks something like this:
from joblib import Parallel, delayed
Parallel(n_jobs=2)(delayed(foo)(i**2) for i in range(10)) # n_jobs = number of processes
The syntax is simple once you understand it. We are using generator syntax in which `delayed` is used to call function `foo` with its arguments contained in the parentheses that follow.
In your case, you should either rewrite your for loop with generator syntax, or define another function (i.e. 'worker' function) to perform the operations of a single loop iteration and place that into the generator syntax of a call to Parallel.
In the later case, you would do something like:
Parallel(n_jobs=2)(delayed(foo)(parameters) for x in range(i,j))
where `foo` is a function you define to handle the body of your for loop. Note that you do not want to append to a list, since Parallel is returning a list anyway.
Problem
Possible Duplicate: how do I parallelize a simple python loop? I'm quite new to Python (using Python 3.2) and I have a question concerning parallelisation. I have a for-loop that I wish to execute in parallel using "multiprocessing" in Python 3.2: ``` def computation: global output for x in range(i,j): localResult = ... #perform some computation as a function of i and j output.append(localResult) ``` In total, I want to perform this computation for a range of i=0 to j=100. Thus I want to create a number of processes that each call the function "computation" with a subdomain of the total range. Any ideas of how do to this? Is there a better way than using multiprocessing? More specific, I want to perform a domain decomposition and I have the following code: ``` from multiprocessing import Pool class testModule: def __init__(self): self def computation(self, args): start, end = args print('start: ', start, ' end: ', end) testMod = testModule() length = 100 np=4 p = Pool(processes=np) p.map(yes tMod.computation, [(length, startPosition, length//np) for startPosition in range(0, length, length//np)]) ``` I get an error message mentioning PicklingError. Any ideas what could be the problem here?