Changing values in pandas dataframe does not work

Question

I’m having a problem changing values in a dataframe. I also want to consult regarding a problem I need to solve and the proper way to use pandas to solve it. I'll appreciate help on both. I have a file containing information about matching degree of audio files to speakers. The file looks something like that: ``` wave_path spk_name spk_example# score mark comments isUsed 190 122_65_02.04.51.800.wav idoD idoD 88 NaN NaN False 191 121_110_20.17.27.400.wav idoD idoD 87 NaN NaN False 192 121_111_00.34.57.300.wav idoD idoD 87 NaN NaN False 193 103_31_18.59.12.800.wav idoD idoD_0 99 HIT VP False 194 131_101_02.08.06.500.wav idoD idoD_0 96 HIT VP False ``` What I need to do, is some kind of a sophisticated counting. I need to group the results by speaker, and calculate for each speaker some calculation. I then proceed with the speaker that made the best calculation for me, but before proceeding I need to mark all the files which I used for the calculation as being used, i.e. changing the isU…

Accepted Answer

Indexing Panda objects can return two fundamentally different objects: a view or a copy.

If `mask` is a basic slice, then `df.ix[mask]` returns a view of `df`. Views share the same underlying data as the original object (`df`). So modifying the view, also modifies the original object.

If `mask` is something more complicated, such as an arbitrary sequence of indices, then `df.ix[mask]` returns a copy of some rows in `df`. Modifying the copy has no affect on the original.

In your case, since the rows which share the same `wave_path` occur at arbitrary locations, `ind_res.ix[example_file]` returns a copy. So

```
ind_res.ix[example_file]['isUsed'] = True
```

has no effect on `ind_res`.

Instead, you could use

```
ind_res.ix[example_file, 'isUsed'] = True
```

to modify `ind_res`. However, see below for a `groupby` suggestion which I think might be closer to what you really want.

Jeff has already provided a link to the Pandas docs which state that

The rules about when a view on the data is returned are entirely dependent on NumPy.

Here are the (complicated) rules which describe when a view or copy is returned. Basically, however, the rule is if the index is requesting a regularly spaced slice of the underlying array then a view is returned, otherwise a copy (out of necessity) is returned.

Here is a simple example which uses basic slice. A view is returned by `df.ix`, so modifying `subdf` modifies `df` as well:

```
import pandas as pd
import numpy as np

df = pd.DataFrame(np.arange(12).reshape(4,3), 
         columns=list('ABC'), index=[0,1,2,3])

subdf = df.ix[0]
print(subdf.values)
# [0 1 2]
subdf.values[0] = 100
print(subdf)
# A    100
# B      1
# C      2
# Name: 0, dtype: int32

print(df)           # df is modified
#      A   B   C
# 0  100   1   2
# 1    3   4   5
# 2    6   7   8
# 3    9  10  11
```

Here is a simple example which uses "fancy indexing" (arbitrary rows selected). A copy is returned by `df.ix`. So modifying `subdf` does not affect `df`.

```
df = pd.DataFrame(np.arange(12).reshape(4,3), 
         columns=list('ABC'), index=[0,1,0,3])

subdf = df.ix[0]
print(subdf.values)
# [[0 1 2]
#  [6 7 8]]

subdf.values[0] = 100
print(subdf)
#      A    B    C
# 0  100  100  100
# 0    6    7    8

print(df)          # df is NOT modified
#    A   B   C
# 0  0   1   2
# 1  3   4   5
# 0  6   7   8
# 3  9  10  11
```

Notice the only difference between the two examples is that in the first, where a view is returned, the index was [0,1,2,3], whereas in the second, where a copy is returned, the index was [0,1,0,3].

Since we are selected rows where the index is 0, in the first example, we can do that with a basic slice. In th second example, the rows where index equals 0 could appear at arbitrary locations, so a copy has to be returned.

Despite having ranted on about the subtlety of Pandas/NumPy slicing, I really don't think that

```
ind_res.ix[example_file, 'isUsed'] = True
```

is what you are ultimately looking for. You probably want to do something more like

```
import pandas as pd
import numpy as np

df = pd.DataFrame(np.arange(12).reshape(4,3), 
                  columns=list('ABC'))
df['A'] = df['A']%2
print(df)
#    A   B   C
# 0  0   1   2
# 1  1   4   5
# 2  0   7   8
# 3  1  10  11

def calculation(grp):
    grp['C'] = True
    return grp

newdf = df.groupby('A').apply(calculation)
print(newdf)
```

which yields

```
   A   B     C
0  0   1  True
1  1   4  True
2  0   7  True
3  1  10  True
```

Changing values in pandas dataframe does not work

Solution

Problem