Using scikit-learn vectorizers and vocabularies with gensim

Question

I am trying to recycle scikit-learn vectorizer objects with gensim topic models. The reasons are simple: first of all, I already have a great deal of vectorized data; second, I prefer the interface and flexibility of scikit-learn vectorizers; third, even though topic modelling with gensim is very fast, computing its dictionaries (`Dictionary()`) is relatively slow in my experience. Similar questions have been asked before, especially here and here, and the bridging solution is gensim's `Sparse2Corpus()` function which transforms a Scipy sparse matrix into a gensim corpus object. However, this conversion does not make use of the `vocabulary_` attribute of sklearn vectorizers, which holds the mapping between words and feature ids. This mapping is necessary in order to print the discriminant words for each topic (`id2word` in gensim topic models, described as "a a mapping from word ids (integers) to words (strings)"). I am aware of the fact that gensim's `Dictionary` objects are much mor…

Accepted Answer

Gensim doesn't require `Dictionary` objects. You can use your plain `dict` as input to `id2word` directly, as long as it maps ids (integers) to words (strings).

In fact anything dict-like will do (including `dict`, `Dictionary`, `SqliteDict`...).

(Btw gensim's `Dictionary` is a simple Python `dict` underneath. Not sure where your remarks on `Dictionary` performance come from, you can't get a mapping much faster than a plain `dict` in Python. Maybe you're confusing it with text preprocessing (not part of gensim), which can indeed be slow.)

Using scikit-learn vectorizers and vocabularies with gensim

Solution

Problem

Related problems