Regression with multi-dimensional targets

python, scikit-learn

Solution

I interpret that what you have is a problem of multiple multivariate regression.

Not every regression method in scikit-learn can handle this sort of problem and you should consult the documentation of each one to find it out. In particular, neither SGDRegressor, GradientBoostingRegressor nor AdaBoostRegressor support this at the moment: `fit(X, y)` specifies X : array-like, shape = [n_samples, n_features] and y: array-like, shape = [n_samples].

However, you can use other methods in scikit-learn. For example, linear models:

from sklearn import linear_model
# multivariate input
X = [[0., 0.], [1., 1.], [2., 2.], [3., 3.]]
# univariate output
Y = [0., 1., 2., 3.]
# multivariate output
Z = [[0., 1.], [1., 2.], [2., 3.], [3., 4.]]

# ordinary least squares
clf = linear_model.LinearRegression()
# univariate
clf.fit(X, Y)
clf.predict ([[1, 0.]])
# multivariate
clf.fit(X, Z)
clf.predict ([[1, 0.]])

# Ridge
clf = linear_model.BayesianRidge()
# univariate
clf.fit(X, Y)
clf.predict ([[1, 0.]])
# multivariate
clf.fit(X, Z)
clf.predict ([[1, 0.]])

# Lasso
clf = linear_model.Lasso()
# univariate
clf.fit(X, Y)
clf.predict ([[1, 0.]])
# multivariate
clf.fit(X, Z)
clf.predict ([[1, 0.]])

Problem

I am using scikit-learn to do regression and my problem is the following. I need to do regression on several parameters (vectors). This works fine with some regression approaches such as `ensemble.ExtraTreesRegressor` and `ensemble.RandomForestRegressor`. Indeed, one can give a vector of vectors as targets to fit the model (`fit(X,y)` method) for the two aforementionned regression methods. However when I try with `ensemble.GradientBoostingRegressor`, `ensemble.AdaBoostRegressor` and `linear_model.SGDRegressor`, the classifier fails to fit the model because it expects 1-dimensional values as targets (y argument of the `fit(X,y)` method). This means, with those Regression methods I can estimate only one parameter at a time. This is not suitable for my problem because it might take some time while I need to estimate about 20 parameters. On the other hand, I really would like to test those approaches. So, my question is: Does anyone know if there is a solution to fit the model once and estimate several parameters for `ensemble.GradientBoostingRegressor`, `ensemble.AdaBoostRegressor` and `linear_model.SGDRegressor`? I hope I've been clear enough ...

Original source