Get subset of most frequent dummy variables in pandas

pandas, python

Solution

use `value_counts()` to do the frequency counting, and then create a mask for the rows that you want remain:

import pandas as pd
values = pd.Series(["a","b","a","b","c","d","e","a"])
counts = pd.value_counts(values)
mask = values.isin(counts[counts > 1].index)
print pd.get_dummies(values[mask])

output:

   a  b
0  1  0
1  0  1
2  1  0
3  0  1
7  1  0

if you want all the data:

values[~mask] = "-"
print pd.get_dummies(values)

output:

   -  a  b
0  0  1  0
1  0  0  1
2  0  1  0
3  0  0  1
4  1  0  0
5  1  0  0
6  1  0  0
7  0  1  0

Problem

I am trying to perform some linear regression analysis, I have some categorical features that i convert to dummy variables using the super awesome get_dummies. The issue I face is, the dataframe gets too big when I add all the elements of the categories. Is there a way (using get_dummies or a more elaborate method) to just create dummy variables of the most frequent terms instead of all of them?

Original source