Reading a UTF8 CSV file with Python

Question

I am trying to read a CSV file with accented characters with Python (only French and/or Spanish characters). Based on the Python 2.5 documentation for the csvreader (http://docs.python.org/library/csv.html), I came up with the following code to read the CSV file since the csvreader supports only ASCII. ``` def unicode_csv_reader(unicode_csv_data, dialect=csv.excel, **kwargs): # csv.py doesn't do Unicode; encode temporarily as UTF-8: csv_reader = csv.reader(utf_8_encoder(unicode_csv_data), dialect=dialect, **kwargs) for row in csv_reader: # decode UTF-8 back to Unicode, cell by cell: yield [unicode(cell, 'utf-8') for cell in row] def utf_8_encoder(unicode_csv_data): for line in unicode_csv_data: yield line.encode('utf-8') filename = 'output.csv' reader = unicode_csv_reader(open(filename)) try: products = [] for field1, field2, field3 in reader: ... ``` Below is an extract of the CSV file I am trying to read: ``` 0665000FS10120684,SD1200IS,Appareil photo numérique PowerShot de 10 Mpx de…

Accepted Answer

The `.encode` method gets applied to a Unicode string to make a byte-string; but you're calling it on a byte-string instead... the wrong way 'round! Look at the `codecs` module in the standard library and `codecs.open` in particular for better general solutions for reading UTF-8 encoded text files. However, for the `csv` module in particular, you need to pass in utf-8 data, and that's what you're already getting, so your code can be much simpler:

```
import csv

def unicode_csv_reader(utf8_data, dialect=csv.excel, **kwargs):
    csv_reader = csv.reader(utf8_data, dialect=dialect, **kwargs)
    for row in csv_reader:
        yield [unicode(cell, 'utf-8') for cell in row]

filename = 'da.csv'
reader = unicode_csv_reader(open(filename))
for field1, field2, field3 in reader:
  print field1, field2, field3 
```

PS: if it turns out that your input data is NOT in utf-8, but e.g. in ISO-8859-1, then you do need a "transcoding" (if you're keen on using utf-8 at the `csv` module level), of the form `line.decode('whateverweirdcodec').encode('utf-8')` -- but probably you can just use the name of your existing encoding in the `yield` line in my code above, instead of `'utf-8'`, as `csv` is actually going to be just fine with ISO-8859-* encoded bytestrings.

Reading a UTF8 CSV file with Python

Solution

Problem

Related problems