How to get the tfidf vector for a given document

tf-idf

Solution

You have a list of documents, 1 to 10. That's 0 to 9 in array-index terms. The variable `tfidx_matrix` will contain a sparse-row form matrix consisting of rows (representing documents) and their normalised association with the vocabulary across the corpus (minus English stop-words).

So to convert the sparse array into a more traditional matrix, you could try

npm_tfidf = tfidf_matrix.todense()
document_1_vector = npm_tfidf[0]
document_2_vector = npm_tfidf[1]
document_3_vector = npm_tfidf[2]
...
document_10_vector = npm_tfidf[9]

There's easier and better ways to extract the contents, but I suppose the part that's getting in your way is this conversion from the sparse matrix representation, that can be tricky to unpick, and the more traditional dense matrix representation.

Note also, that interpreting the vectors will require you being able to extract the vocabulary extracted during the process - this should be in the form of an ordered (alphabetically list of tokens) and can be extracted using:

vocabulary = tfidf_matrix.get_feature_names()

Problem

I have the following file: ``` id review 1 "Human machine interface for lab abc computer applications." 2 "A survey of user opinion of computer system response time." 3 "The EPS user interface management system." 4 "System and human system engineering testing of EPS." 5 "Relation of user perceived response time to error measurement." 6 "The generation of random binary unordered trees." 7 "The intersection graph of paths in trees." 8 "Graph minors IV Widths of trees and well quasi ordering." 9 "Graph minors A survey." 10 "survey is a state of art." ``` Each row concern a document. I convert these documents to a corpus and i find for each word its TFIDF: ``` from collections import defaultdict import csv from sklearn.feature_extraction.text import TfidfVectorizer reviews = defaultdict(list) with open("C:/Users/user/workspacePython/Tutorial/data/unlabeledTrainData.tsv", "r") as sentences_file: reader = csv.reader(sentences_file, delimiter='\t') reader.next() for row in reader: reviews[row[1]].append(row[1]) for id, review in reviews.iteritems(): reviews[id] = " ".join(review) corpus = [] for id, review in sorted(reviews.iteritems(), key=lambda t: id): corpus.append(review) tf = TfidfVectorizer(analyzer='word', ngram_range=(1,1), min_df = 1, stop_words = 'english') tfidf_matrix = tf.fit_transform(corpus) ``` My question is: How can i get for a given document(from the above file) its correponding vector(row) in the tfidf_matrix. Thank you

Original source