How to filter out words with low tf-idf in a corpus with gensim?
gensim, nlp, python
Solution
Say your corpus is the following:
corpus = [dictionary.doc2bow(doc) for doc in documents]
After running TFIDF you can retrieve a list of low value words:
tfidf = TfidfModel(corpus, id2word=dictionary)
low_value = 0.2
low_value_words = []
for bow in corpus:
low_value_words += [id for id, value in tfidf[bow] if value < low_value]
Then filter them out of the dictionary before running LDA:
dictionary.filter_tokens(bad_ids=low_value_words)
Recompute the corpus now that low value words are filtered out:
new_corpus = [dictionary.doc2bow(doc) for doc in documents]
Problem
I am using `gensim` for some NLP task. I've created a corpus from `dictionary.doc2bow` where `dictionary` is an object of `corpora.Dictionary`. Now I want to filter out the terms with low tf-idf values before running an LDA model. I looked into the documentation of the corpus class but cannot find a way to access the terms. Any ideas? Thank you.