groupby, apply, and set not behaving as expected... is this a bug?

Question

To see the issue, consider the following dataframe ``` In [66]: dat = pandas.DataFrame(['a','b','c','d','e','f','g','h'], columns=['letters']) In [67]: dat['numbers'] = pandas.Series([1,2,3,4,5,6,7,8]) In [68]: dat['names'] = pandas.Series(['jim','jan','jerry','george' ,'mary','mary','sue','sue']) In [69]: dat Out[69]: letters numbers names 0 a 1 jim 1 b 2 jan 2 c 3 jerry 3 d 4 george 4 e 5 mary 5 f 6 mary 6 g 7 sue 7 h 8 sue ``` Group by names ``` In [78]: dat = dat.groupby(['names'])[['letters']] ``` Now my attempt to concat letters produces a funny result: ``` In [80]: dat.apply(lambda x: '|'.join(set(x))) Out[80]: names george letters|numbers|names jan letters|numbers|names jerry letters|numbers|names jim letters|numbers|names mary letters|numbers|names sue letters|numbers|names dtype: object ``` The following hack seems to work but why do I need to select 'letters' again, and why does the output above look as it does? ``` In [84]: dat.apply(lambda x: '|'.join(set(x['letters'])))…

Accepted Answer

It may seem a bit strange, but as you see set of a DataFrame is it's columns:

```
In [11]: dat
Out[11]:
  letters  numbers   names
0       a        1     jim
1       b        2     jan
2       c        3   jerry
3       d        4  george
4       e        5    mary
5       f        6    mary
6       g        7     sue
7       h        8     sue

[8 rows x 3 columns]

In [12]: set(dat)
Out[12]: {'letters', 'names', 'numbers'}
```

That's due to the way you iterate through a DataFrame (by the columns):

```
In [13]: for i in dat: print(i)
letters
numbers
names
```

This would work with the SeriesGroupBy (iterating through a Series iterates through its elements):

```
In [21]: g = dat.groupby(['names'])['letters']

In [22]: g.apply(lambda x: '|'.join(set(x)))
Out[22]:
names
george      d
jan         b
jerry       c
jim         a
mary      e|f
sue       h|g
dtype: object
```

Note: You don't need the set or, indeed, the lambda:

```
In [23]: g.apply('|'.join)
Out[23]:
names
george      d
jan         b
jerry       c
jim         a
mary      e|f
sue       g|h
dtype: object
```

groupby, apply, and set not behaving as expected... is this a bug?

Solution

Problem