UnicodeEncodeError when writing to CSV from JSON

csv, encoding, json, python

Solution

Explicitly encode the field. Otherwise Python try to encode it using ascii encoding.

>>> print u'\u2026'.encode('ascii')

Traceback (most recent call last):
  File "<pyshell#2>", line 1, in <module>
    print u'\u2026'.encode('ascii')
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2026' in position 0: ordinal not in range(128)
>>> print u'\u2026'.encode('utf-8')
…

f.writerow([
    status['id'],
    status['text'].encode('utf-8'), # <----
    status['created_at'],
    status['coordinates'],
    status['user']['id_str'],
    status['retweet_count'],
])

Problem

I'm getting the following error when attempting to write to a CSV using JSON: ``` Traceback (most recent call last): File "twitter_search_csv.py", line 25, in <module> status['retweet_count'], UnicodeEncodeError: 'ascii' codec can't encode character u'\u2026' in position 139: ordinal not in range(128) ``` Here's the code I'm working with: ``` import requests import urllib2 from requests_oauthlib import OAuth1 import csv auth = OAuth1('', '', '', '') url = 'https://api.twitter.com/1.1/search/tweets.json?q=%23OpeningCeremony' response = requests.get(url, auth=auth) data = response.json()['statuses'] with open('olympic_search.csv', 'wb') as csvfile: f = csv.writer(csvfile) for status in data: f.writerow([ status['id'], status['text'], status['created_at'], status['coordinates'], status['user']['id_str'], status['retweet_count'], ]) ```

Original source