Bag of Words representation problem

Question

Basically i have a dictionary containing all the words of my vocabulary as keys, and all with 0 as value. To process a document into a bag of words representation i used to copy that dictionary with the appropriate IEqualityComparer and simply checked if the dictionary contained every word in the document and incremented it's key. To get the array of the bag of words representation i simply used the ToArray method. This seemed to work fine, but i was just told that the dictionary doesnt assure the same Key order, so the resulting arrays might represent the words in different order, making it useless. My current idea to solve this problem is to copy all the keys of the word dictionary into an ArrayList, create an array of the proper size and then use the indexOf method of the array list to fill the array. So my question is, is there any better way to solve this, mine seems kinda crude... and won't i have issues because of the IEqualityComparer?

Accepted Answer

Let me see if I understand the problem. You have two documents D1 and D2 each containing a sequence of words drawn from a known vocabulary {W1, W2... Wn}. You wish to obtain two mappings indicating the number of occurrences of each word in each document. So for D1, you might have

```
W1 --> 0
W2 --> 1
W3 --> 4
```

indicating that D1 was perhaps "W3 W2 W3 W3 W3". Perhaps D2 is "W2 W1 W2", so its mapping is

```
W1 --> 1
W2 --> 2
W3 --> 0
```

You wish to take both mappings and determine the vectors [0, 1, 4] and [1, 2, 0] and then compute the angle between those vectors as a way of determining how similar or different the two documents are.

Your problem is that the dictionary does not guarantee that the key/value pairs are enumerated in any particular order.

OK, so order them.

```
vector1 = (from pair in map1 orderby pair.Key select pair.Value).ToArray();
vector2 = (from pair in map2 orderby pair.Key select pair.Value).ToArray();
```

and you're done.

Does that solve your problem, or am I misunderstanding the scenario?

Bag of Words representation problem

Solution

Problem