Efficient lookup by common words

Question

I have a list of names (strings) divided into words. There are 8 million names, each name consists of up to 20 words (tokens). Number of unique tokens is 2.2 million. I need an efficient way to find all names containing at least one word from the query (which may contain also up to 20 words, but usually only a few). My current approach uses Python Pandas and looks like this (later referred as `original`): ``` >>> df = pd.DataFrame([['foo', 'bar', 'joe'], ['foo'], ['bar', 'joe'], ['zoo']], index=['id1', 'id2', 'id3', 'id4']) >>> df.index.rename('id', inplace=True) # btw, is there a way to include this into prev line? >>> print df 0 1 2 id id1 foo bar joe id2 foo None None id3 bar joe None id4 zoo None None def filter_by_tokens(df, tokens): # search within each column and then concatenate and dedup results results = [df.loc[lambda df: df[i].isin(tokens)] for i in range(df.shape[1])] return pd.concat(results).reset_index().drop_duplicates().set_index(df.index.name) >>> print filter_by_to…

Accepted Answer

idea 0 `zip`

```
def pir(s, token):
    return s[[bool(p & token) for p in s]]

pir(s, {'foo', 'zoo'})
```

idea 1 `merge`

```
token = pd.DataFrame(dict(v=['foo', 'zoo']))
d1 = df.stack().reset_index('id', name='v')
s.ix[d1.merge(token).id.unique()]
```

idea 2 `mul` + `any`

```
d1 = pd.get_dummies(df.stack()).groupby(level=0).sum()
token = pd.Series(1, ['foo', 'zoo'])
s[d1.mul(token).any(1)]
```

idea 3 `isin`

```
d1 = df.stack()
s[d1.isin({'zoo', 'foo'}).unstack().any(1)]
```

idea 4 `query`

```
token = ('foo', 'zoo')
d1 = df.stack().to_frame('s')
s.ix[d1.query('s in @token').index.get_level_values(0).unique()]
```

Efficient lookup by common words

Solution

Problem