Scikit-Learn Logistic Regression Memory Error

Question

I'm attempting to use sklearn 0.11's LogisticRegression object to fit a model on 200,000 observations with about 80,000 features. The goal is to classify short text descriptions into 1 of 800 classes. When I attempt to fit the classifier pythonw.exe gives me: Application Error "The instruction at ... referenced memory at 0x00000000". The memory could not be written". The features are extremely sparse, about 10 per observation, and are binary (either 1 or 0), so by my back of the envelope calculation my 4 GB of RAM should be able to handle the memory requirements, but that doesn't appear to be the case. The models only fit when I use fewer observations and/or fewer features. If anything, I would like to use even more observations and features. My naive understanding is that the liblinear library running things behind the scenes is capable of supporting that. Any ideas for how I might squeeze a few more observations in? My code looks like this: ``` y_vectorizer = LabelVectorizer(y) # my…

Accepted Answer

liblinear (the backing implementation of `sklearn.linear_model.LogisticRegression`) will host its own copy of the data because it is a C++ library whose internal memory layout cannot be directly mapped onto a pre-allocated sparse matrix in scipy such as `scipy.sparse.csr_matrix` or `scipy.sparse.csc_matrix`.

In your case I would recommend to load your data as a `scipy.sparse.csr_matrix` and feed it to a `sklearn.linear_model.SGDClassifier` (with `loss='log'` if you want a logistic regression model and the ability to call the `predict_proba` method). `SGDClassifier` will not copy the input data if it's already using the `scipy.sparse.csr_matrix` memory layout.

Expect it to allocate a dense model of 800 * (80000 + 1) * 8 / (1024 ** 2) = 488MB in memory (in addition to the size of your input dataset).

Edit: how to optimize the memory access for your dataset

To free memory after dataset extraction you can:

```
x_vectorizer = CountVectorizer(binary = True, analyzer = features)
x = x_vectorizer.fit_transform(x)
from sklearn.externals import joblib
joblib.dump(x.tocsr(), 'dataset.joblib')
```

Then quit this python process (to force complete memory deallocation) and in a new process:

```
x_csr = joblib.load('dataset.joblib')
```

Under linux / OSX you could memory map that even more efficiently with:

```
x_csr = joblib.load('dataset.joblib', mmap_mode='c')
```

Scikit-Learn Logistic Regression Memory Error

Solution

Problem