Using Sklearn's TfidfVectorizer transform

Question

I am trying to get the tf-idf vector for a single document using Sklearn's TfidfVectorizer object. I create a vocabulary based on some training documents and use fit_transform to train the TfidfVectorizer. Then, I want to find the tf-idf vectors for any given testing document. ``` from sklearn.feature_extraction.text import TfidfVectorizer self.vocabulary = "a list of words I want to look for in the documents".split() self.vect = TfidfVectorizer(sublinear_tf=True, max_df=0.5, analyzer='word', stop_words='english') self.vect.fit_transform(self.vocabulary) ... doc = "some string I want to get tf-idf vector for" tfidf = self.vect.transform(doc) ``` The problem is that this returns a matrix with n rows where n is the size of my doc string. I want it to return just a single vector representing the tf-idf for the entire string. How can I make this see the string as a single document, rather than each character being a document? Also, I am very new to text mining so if I am doing something w…

Accepted Answer

If you want to compute tf-idf only for a given vocabulary, use `vocabulary` argument to `TfidfVectorizer` constructor,

```
vocabulary = "a list of words I want to look for in the documents".split()
vect = TfidfVectorizer(sublinear_tf=True, max_df=0.5, analyzer='word', 
           stop_words='english', vocabulary=vocabulary)
```

Then, to fit, i.e. calculate counts, with a given `corpus`, i.e. an iterable of documents, use `fit`:

```
vect.fit(corpus)
```

Method `fit_transform` is a shortening for

```
vect.fit(corpus)
corpus_tf_idf = vect.transform(corpus) 
```

Last, `transform` method accepts a corpus, so for a single document, you should pass it as list, or it is treated as iterable of symbols, each symbol being a document.

```
doc_tfidf = vect.transform([doc])
```

Using Sklearn's TfidfVectorizer transform

Solution

Problem