Remove duplicate rows in pandas dataframe based on condition
pandas, python
Solution
You can use groupby on index after sorting the df by valu.
df.sort_values(by='valu', ascending=False).groupby(level=0).first()
Out[1277]:
is_avail valu data_source
2015-08-07 False 0.582 source_b
2015-08-23 False 0.296 source_a
2015-09-08 False 0.433 source_a
2015-10-01 True 0.169 source_b
Problem
``` is_avail valu data_source 2015-08-07 False 0.282 source_a 2015-08-07 False 0.582 source_b 2015-08-23 False 0.296 source_a 2015-09-08 False 0.433 source_a 2015-10-01 True 0.169 source_b ``` In the dataframe above, I want to remove the duplicate rows (i.e. row where the index is repeated) by retaining the row with a higher value in the `valu` column. I can remove rows with duplicate indexes like this: `df = df[~df.index.duplicated()]`. But how to remove based on condition specified above?