Using Scikit's LabelEncoder correctly across multiple programs

Question

The basic task that I have at hand is a) Read some tab separated data. b) Do some basic preprocessing c) For each categorical column use `LabelEncoder` to create a mapping. This is don somewhat like this ``` mapper={} #Converting Categorical Data for x in categorical_list: mapper[x]=preprocessing.LabelEncoder() for x in categorical_list: df[x]=mapper[x].fit_transform(df.__getattr__(x)) ``` where `df` is a pandas dataframe and `categorical_list` is a list of column headers that need to be transformed. d) Train a classifier and save it to disk using `pickle` e) Now in a different program, the model saved is loaded. f) The test data is loaded and the same preprocessing is performed. g) The `LabelEncoder's` are used for converting categorical data. h) The model is used to predict. Now the question that I have is, will the step `g)` work correctly? As the documentation for `LabelEncoder` says ``` It can also be used to transform non-numerical labels (as long as they are hashable and compar…

Accepted Answer

According to the `LabelEncoder` implementation, the pipeline you've described will work correctly if and only if you `fit` LabelEncoders at the test time with data that have exactly the same set of unique values.

There's a somewhat hacky way to reuse LabelEncoders you got during train. `LabelEncoder` has only one property, namely, `classes_`. You can pickle it, and then restore like

Train:

```
encoder = LabelEncoder()
encoder.fit(X)
numpy.save('classes.npy', encoder.classes_)
```

Test

```
encoder = LabelEncoder()
encoder.classes_ = numpy.load('classes.npy')
# Now you should be able to use encoder
# as you would do after `fit`
```

This seems more efficient than refitting it using the same data.

Using Scikit's LabelEncoder correctly across multiple programs

Solution

Problem