Multithreaded calls to the objective function of scipy.optimize.leastsq

mathematical-optimization, python, scipy

Solution

There's a good opportunity to speed up `leastsq` by supplying your own function to calculate the derivatives (the `Dfun` parameter), providing you have several parameters. If this function is not supplied, `leastsq` iterates over each of the parameters to calculate the derivative each time, which is time consuming. This appears to take the majority of the time in the fitting.

You can use your own `Dfun` function which calculates the derivatives for each parameter using a `multiprocessing.Pool` to do the work. These derivatives can be calculated independently and should be trivially parallelised.

Here is a rough example, showing how to do this:

import numpy as np
import multiprocessing
import scipy.optimize

def calcmod(params):
    """Return the model."""
    return func(params)

def delta(params):
    """Difference between model and data."""
    return calcmod(params) - y

pool = multiprocessing.Pool(4)

def Dfun(params):
    """Calculate derivatives for each parameter using pool."""
    zeropred = calcmod(params)

    derivparams = []
    delta = 1e-4
    for i in range(len(params)):
        copy = np.array(params)
        copy[i] += delta
        derivparams.append(copy)

    results = pool.map(calcmod, derivparams)
    derivs = [ (r - zeropred)/delta for r in results ]
    return derivs

retn = scipy.optimize.leastsq(leastfuncall, inputparams, gtol=0.01,
                              Dfun=Dfun, col_deriv=1)

Problem

I'm using `scipy.optimize.leastsq` in conjunction with a simulator. `leastsq` calls a user-defined objective function and passes an input vector to it. In turn, the objective function returns an error vector. `leastsq` optimizes the input vector in such a way that the sum of the squares of the error vector is minimized. In my case the objective function will run a whole simulation each time it is called. The employed simulator is single-threaded and needs several minutes for each run. I'd therefore like to run multiple instances of the simulator at once. However, calls to the objective function are performed serially. How can I get `leastsq` to perform multiple calls to the objective function at once?

Original source