tfidf algorithm for python

Question

I have this code for calculating text similarity with tf-idf. ``` from sklearn.feature_extraction.text import TfidfVectorizer documents = [doc1,doc2] tfidf = TfidfVectorizer().fit_transform(documents) pairwise_similarity = tfidf * tfidf.T print pairwise_similarity.A ``` The problem is that this code take as an input plain strings and I want to prepare the documents by removing stopwords, stemming and tokkenize. So the input would be a list. The error if I call the `documents = [doc1,doc2]` with the tokkenized documents is: ``` Traceback (most recent call last): File "C:\Users	asos\Desktop\my thesis\beta\similarity.py", line 18, in <module> tfidf = TfidfVectorizer().fit_transform(documents) File "C:\Python27\lib\site-packages\scikit_learn-0.14.1-py2.7-win32.egg\sklearn\feature_extraction	ext.py", line 1219, in fit_transform X = super(TfidfVectorizer, self).fit_transform(raw_documents) File "C:\Python27\lib\site-packages\scikit_learn-0.14.1-py2.7-win32.egg\sklearn\feature_extraction	…

Accepted Answer

Try skipping preprocessing to lowercase and supply your own "nop" tokenizer:

```
tfidf = TfidfVectorizer(tokenizer=lambda doc: doc, lowercase=False).fit_transform(documents)
```

You should also check out other parameters like `stop_words` to avoid duplicating your preprocessing.

tfidf algorithm for python

Solution

Problem