How to parallelize a for in python inside a class?

Question

I have a python function `funz` that returns every time a different array of length p. I need to run this function different times and then to compute the mean of each value. I can do this with a for loop but it takes a lot of times. I am trying to use the library multiprocessing but I get into an error. ``` import sklearn as sk import numpy as np from sklearn.base import BaseEstimator, TransformerMixin from sklearn import preprocessing,linear_model, cross_validation from scipy import stats from multiprocessing import Pool class stabilize(BaseEstimator,TransformerMixin): def __init__(self,sim=3,n_folds=3): self.sim=sim self.n_folds=n_folds def fit(self,X,y): self.n,self.p=X.shape self.X=X self.y=y self.beta=np.zeros(shape=(self.sim,self.p)) self.alpha_min=[] self.mapper=p.map(self.multiple_cv,[1]*self.sim) def multiple_cv(self,o): kf=sk.cross_validation.KFold(self.n,n_folds=self.n_folds,shuffle=True) cv=sk.linear_model.LassoCV(cv=kf).fit(self.X,self.y) beta=cv.coef_ alpha_min=cv.alpha…

Accepted Answer

Your problem is that the function you want to call is a instance method of an object. This can not be serialized and sent to another process. I see two solutions:

use a different globally available function:

```
class stabilize(BaseEstimator,TransformerMixin):
    ...
def multiple_cv((self,o)):
    ...
```

and

```
    self.mapper=p.map(self.multiple_cv,[(self, 1)]*self.sim)
```

make the methods of objects serializable using VeryPicklableObject and its dependencies.

```
    @picklableInstancemethod
    def multiple_cv(self, o):
        ...
```

How to parallelize a for in python inside a class?

Solution

Problem