Parallel mapping functions in IPython w/ multiple parameters

Question

I'm trying to use IPython's parallel environment and so far, it's looking great but I'm running into a problem. Lets say that I have a function, defined in a library ``` def func(a,b): ... ``` that I use when I want to evaluate on one value of a and a bunch of values of b. ``` [func(myA, b) for b in myLongList] ``` Obviously, the real function is more complicated but the essence of the matter is that it takes multiple parameters and I'd like to map over only one of them. The problem is that map, @dview.parallel, etc. map over all the arguments. So lets say I want to get the answer to func(myA, myLongList). The obvious way to do this is to curry, either w/ functools.partial or just as ``` dview.map_sync(lambda b: func(myA, b), myLongList) ``` However, this does not work correctly on remote machines. The reason is that when the lambda expression is pickled, the value of myA is not included and instead, the value of myA from the local scope on the remote machine is used. When closures ge…

Accepted Answer

I can improve a bit on batu's answer (which I think is a good one, but doesn't perhaps document in as much detail WHY you use those options). The ipython documentation is also currently woefully inadequate on this point. So your function is of the form:

```
def myfxn(a,b,c,d):
  ....
  return z
```

and stored in a file called mylib. Lets say b,c, and d are the same during your run, so you write a lambda function to reduce it to a 1-parameter function.

```
import mylib
mylamfxn=lambda a:mylib.myfxn(a,b,c,d)
```

and you want to run:

```
z=dview.map_sync(mylamfxn, iterable_of_a)
```

In a dream world, everything would magically work like that. However, first you'd get an error of "mylib not found," because the ipcluster processes haven't loaded mylib. Make sure the ipcluster processes have "mylib" in their python path and are in the correct working directory for myfxn, if necessary. Then you need to add to your python code:

```
dview.execute('import mylib')
```

which runs the `import mylib` command on each process. If you try again, you'll get an error along the lines of "global variable b not defined" because while the variables are in your python session, they aren't in the ipcluster processes. However, python provides a method of copying a group of variables to the subprocesses. Continuing the example above:

```
mydict=dict(b=b, c=c, d=d)
dview.push(mydict)
```

Now all of the subprocesses have access to b,c,and d. Then you can just run:

```
z=dview.map_sync(mylamfxn, iterable_of_a)
```

and it should now work as advertised. Anyway, I'm new to parallel computing with python, and found this thread useful, so I thought I'd try to help explain a few of the points that confused me a bit....

The final code would be:

```
import mylib

#set up parallel processes, start ipcluster from command line prior!
from IPython.parallel import Client
rc=Client()
dview=rc[:]

#...do stuff to get iterable_of_a and b,c,d....

mylamfxn=lambda a:mylib.myfxn(a,b,c,d)

dview.execute('import mylib')
mydict=dict(b=b, c=c, d=d)
dview.push(mydict)
z=dview.map_sync(mylamfxn, iterable_of_a)
```

This is probably the quickest and easiest way to make pretty much any embarrassingly parallel code run parallel in python....

UPDATE You can also use dview to push all the data without loops and then use an lview (i.e. `lview=rc.load_balanced_view(); lview.map(...)` to do the actual calculation in load balanced fashion.

Parallel mapping functions in IPython w/ multiple parameters

Solution

Problem