Pandas returning empty groups in groupby

Question

I have a Pandas DataFrame with 3 columns, `target`, `pred`, and `conf_bin`. If I run a `groupby(by='conf_bin').apply(...)` my apply function gets called with empty `DataFrame`s for values that do not appear in the `conf_bin` column. How is this possible? Details The DataFrame looks something like this: ``` target pred conf_bin 0 5 6 0.50 1 4 4 0.60 2 4 4 0.50 3 4 3 0.50 4 4 5 0.50 5 5 5 0.55 6 5 5 0.55 7 5 5 0.55 ``` Obviously `conf_bin` is a numeric bin with values in the range `np.arange(0, 1, 0.05)`. However, not all values are present in the data: ``` In [224]: grp = tp.groupby(by='conf_bin') In [225]: grp.groups.keys() Out[225]: dict_keys([0.5, 0.60000000000000009, 0.35000000000000003, 0.75, 0.85000000000000009, 0.65000000000000002, 0.55000000000000004, 0.80000000000000004, 0.20000000000000001, 0.45000000000000001, 0.40000000000000002, 0.30000000000000004, 0.70000000000000007, 0.25]) ``` So, for example, the values `0` and `0.05` do not appear. However, when I run an `apply` on t…

Accepted Answer

I too was having this problem, which popped up when trying to create subplots for every category in my dataframe.

I came up with the following workaround (based on this SO post), by pulling out the non-empty groups into a list.

```
groups = df.groupby('conf_bin')
group_list = [(index, group) for index, group in groups if len(group) > 0]
```

It does break the implicit contract that "you wrangle your data in pandas", and probably mismanages memory, but it works.

Now you can iterate through your groupby list with the same interface as with a groupby object, e.g.

```
fig, axes = plt.subplots(nrows=len(group_list), ncols=1)
for (index, group), ax in zip(group_list, axes.flatten()):
    group['target'].plot(ax=ax, title=index)
```

Pandas returning empty groups in groupby

Solution

Problem

Related problems