How to remove duplicates from a csv file

Question

I have downloaded a CSV file from Hotmail, but it has a lot of duplicates in it. These duplicates are complete copies and I don't know why my phone created them. I want to get rid of the duplicates. Technical specification: ``` Windows XP SP 3 Python 2.7 CSV file with 400 contacts ```

Accepted Answer

UPDATE: 2016

If you are happy to use the helpful `more_itertools` external library:

```
from more_itertools import unique_everseen
with open('1.csv', 'r') as f, open('2.csv', 'w') as out_file:
    out_file.writelines(unique_everseen(f))
```

A more efficient version of @IcyFlame's solution

```
with open('1.csv', 'r') as in_file, open('2.csv', 'w') as out_file:
    seen = set() # set for fast O(1) amortized lookup
    for line in in_file:
        if line in seen: continue # skip duplicate

seen.add(line)
        out_file.write(line)
```

To edit the same file in-place you could use this (Old Python 2 code)

```
import fileinput
seen = set() # set for fast O(1) amortized lookup
for line in fileinput.FileInput('1.csv', inplace=1):
    if line in seen: continue # skip duplicate

seen.add(line)
    print line, # standard output is now redirected to the file
```

How to remove duplicates from a csv file

Solution

Problem

Related problems