Merge two DataFrames with some equal columns
csv, pandas, python
Solution
Usually you can solve this with the proper index:
df1.set_index(['id', 'noteId'], inplace=True)
df1.update(df2)
(And if you don't want that index after, just `df1.reset_index(inplace=True)`)
Problem
I have two csv files: 1.csv ``` id,noteId,text id2,idNote19,This is my old text 2 id5,idNote13,This is my old text 5 id1,idNote12,This is my old text 1 id3,idNote10,This is my old text 3 id4,idNote11,This is my old text 4 ``` 2.csv ``` id,noteId,text,other id3,idNote10,new text 3,On1 id2,idNote19,My new text 2,Pre8 ``` Loading them like: ``` >>> df1 = pd.read_csv('1.csv', encoding='utf-8').set_index('id') >>> df2 = pd.read_csv('2.csv', encoding='utf-8').set_index('id') >>> >>> print df1 noteId text id id2 idNote19 This is my old text 2 id5 idNote13 This is my old text 5 id1 idNote12 This is my old text 1 id3 idNote10 This is my old text 3 id4 idNote11 This is my old text 4 >>> print df2 noteId text other id id3 idNote10 new text 3 On1 id2 idNote19 My new text 2 Pre8 id5 NaN My new text 2 Hl0 id22 idNote22 My new text 22 M1 ``` I need merge both DataFrames in something like this (ovewriting values on df1 that are empty on df2, adding extra columns and rows that not exists on df1): ``` noteId text other id id2 idNote19 My new text 2 Pre8 id5 NaN My new text 2 Hl0 id1 idNote12 This is my old text 1 NaN id3 idNote10 new text 3 On1 id4 idNote11 This is my old text 4 NaN id22 idNote22 My new text 22 M1 ``` My real DataFrames has other columns that should be merged too, not just `text` I tried using `merge` getting something like: ``` >>> df1 = pd.read_csv('1.csv', encoding='utf-8') >>> df2 = pd.read_csv('2.csv', encoding='utf-8') >>> >>> print df1 id noteId text 0 id2 idNote19 This is my old text 2 1 id5 idNote13 This is my old text 5 2 id1 idNote12 This is my old text 1 3 id3 idNote10 This is my old text 3 4 id4 idNote11 This is my old text 4 >>> print df2 id noteId text 0 id3 idNote10 new text 3 1 id2 idNote19 My new text 2 >>> >>> print merge(df1, df2, how='left', on=['id']) id noteId_x text_x noteId_y text_y 0 id2 idNote19 This is my old text 2 idNote19 My new text 2 1 id5 idNote13 This is my old text 5 NaN NaN 2 id1 idNote12 This is my old text 1 NaN NaN 3 id3 idNote10 This is my old text 3 idNote10 new text 3 4 id4 idNote11 This is my old text 4 NaN NaN >>> ``` But it's not what I need. I don't know if I'm on the right path and should merge the suffixed columns or if there is a better way to do this. Thanks! Update: Added ovewriting values on df1 that are empty on df2, adding extra columns on df2 that should be present on df1 after "merge" and rows that should be appended on df1 -- SOLUTION Based on a @U2EF1 (thanks!) comment, I found the solution: ``` df1.fillna(value='None', inplace=True) df2.fillna(value='None', inplace=True) concat([df1, df2]).groupby('id').last().fillna(value='None') ``` In my case, it's very important to define a default "empty" value, that's why the `fillna`.