Scikit learn - fit_transform on the test set

Question

I am struggling to use Random Forest in Python with Scikit learn. My problem is that I use it for text classification (in 3 classes - positive/negative/neutral) and the features that I extract are mainly words/unigrams, so I need to convert these to numerical features. I found a way to do it with `DictVectorizer`'s `fit_transform`: ``` from sklearn.preprocessing import LabelEncoder from sklearn.metrics import classification_report from sklearn.feature_extraction import DictVectorizer vec = DictVectorizer(sparse=False) rf = RandomForestClassifier(n_estimators = 100) trainFeatures1 = vec.fit_transform(trainFeatures) # Fit the training data to the training output and create the decision trees rf = rf.fit(trainFeatures1.toarray(), LabelEncoder().fit_transform(trainLabels)) testFeatures1 = vec.fit_transform(testFeatures) # Take the same decision trees and run on the test data Output = rf.score(testFeatures1.toarray(), LabelEncoder().fit_transform(testLabels)) print "accuracy: " + str(Outpu…

Accepted Answer

You are not supposed to do `fit_transform` on your test data, but only `transform`. Otherwise, you will get different vectorization than the one used during training.

For the memory issue, I recommend `TfIdfVectorizer`, which has numerous options of reducing the dimensionality (by removing rare unigrams etc.).

UPDATE

If the only problem is fitting test data, simply split it to small chunks. Instead of something like

```
x=vect.transform(test)
eval(x)
```

you can do

```
K=10
for i in range(K):
    size=len(test)/K
    x=vect.transform(test[ i*size : (i+1)*size ])
    eval(x)
```

and record results/stats and analyze them afterwards.

in particular

```
predictions = []

K=10
for i in range(K):
    size=len(test)/K
    x=vect.transform(test[ i*size : (i+1)*size ])
    predictions += rf.predict(x) # assuming it retuns a list of labels, otherwise - convert it to list

print accuracy_score( predictions, true_labels )
```

Scikit learn - fit_transform on the test set

Solution

Problem