python and scrapy THE encoding issue

Question

I simple can't figure out! :( I am scrapping data from an utf-8 encoded site, well that is at least what it says: ``` Content-Type: text/html;charset=utf-8 ``` I am getting a list of regular unicode strings with XPath selector extract() call: ``` item['city']= element.select('//div[@id="bubble_2"]/div/text()').extract() ``` This is the list: ``` [u'Westbahnhofstr.\xa010', u'72070\xa0T\xfcbingen'] ``` Now I join the list into one unicode string: ``` item['city']= "".join(element.select('//div[@id="bubble_2"]/div/text()').extract()) ``` So far so good: ``` u'Beim Nonnenhaus\xa0672070\xa0T\xfcbingen' ``` The problem appears while I try to output this unicode string either to screen (print) or to a file (write). whatever I try it returns an error (http://pastebin.com/51DkX2R2): ``` exceptions.UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 11: ordinal not in range(128) ``` I have encoded unicode to byte string before output of course: ``` item['city'].encode('…

Accepted Answer

You are ignoring the result of the `.encode()` call:

```
item['city'].encode('utf-8')
```

Strings are immutable, and are not encoded in-place. Even better, the type of the returned object is different. You'll need to assign the return value back:

```
item['city'] = item['city'].encode('utf-8')
```

However, you should not use `codecs.open()` for the CSV file. The `csv` module will always write bytestrings, not Unicode.

By using a `codecs.open()` file object, an implicit decode takes place to get back to Unicode, and it is that that fails for you; it is why you get a `UnicodeDecodeError` exception, not an encode error:

```
  File "C:\Python27\lib\codecs.py", line 351, in write
    data, consumed = self.encode(object, self.errors)
exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 11: ordinal not in range(128)
```

Use a regular `open()` call instead:

```
self.modelsCsv = csv.writer(open('Dees.csv', mode='wb'))
```

Note the `'wb'`; the `csv` module handles line endings itself.

python and scrapy THE encoding issue

Solution

Problem