Pandas groupby apply function that combines some groups but not others

Question

I'm using pandas `groupby` on my DataFrame `df` which has columns `type`, `subtype`, and 11 others. I'm then calling an `apply` with my `combine_function` (needs a better name) on the groups like: ``` grouped = df('type') reduced = grouped.apply(combine_function) ``` where my `combine_function` checks if any element in the group contains any element with the given subtype, say 1, and looks like: ``` def combine_function(group): if 1 in group.subtype: return aggregate_function(group) else: return group ``` The `combine_function` then can call an `aggregate_function`, that calculates summary statistics, stores them in the first row, and then sets that row to be the group. It looks like: ``` def aggregate_function(group): first = group.first_valid_index() group.value1[group.index == first] = group.value1.mean() group.value2[group.index == first] = group.value2.max() group.value3[group.index == first] = group.value3.std() group = group[(group.index == first)] return group ``` I'm fairly s…

Accepted Answer

Your `aggregate_functions` looks contorted to me. When you aggregate a group, it automatically reduces to one row; you don't need to do it manually. Maybe I am missing the point. (Are you doing something special with the index that I'm not understanding?) But a more normal usage would look like this:

```
agg_condition = lambda x: Series([1]).isin(x['subtype]').any()
agg_functions = {'value1': np.mean, 'value2': np.max, 'value3': np.std}

df1 = df.groupby('type').filter(agg_condition).groupby('type').agg(**agg_functions)
df2 = df.groupby('type').filter(~agg_condition)

result = pd.concat([df1, df2])
```

Note: `agg_condition` is messy because (1) built-in Python `in` refers to the index of a Series, not its values, and (2) the result has to be reduced to a scalar by `any()`.

Pandas groupby apply function that combines some groups but not others

Solution

Problem