Converting to UTF-8 (again)

Question

I've this string `Traor\u0102\u0160` `Traor\u0102\u0160` Should produce `TraorÃ©`. Then `TraorÃ©` utf-8 decoded should produce `Traorè` How I can convert it to `Traorè` ? What kind of chars are `Traor\u0102\u0160`? Unicode? I've already read this http://docs.python.org/howto/unicode.html#encodings many times. But I'm still really confused. I get this data with the following request: ``` import json import requests # making a request to get this json r = requests.get('http://cdn.content.easports.com/fifa/fltOnlineAssets/2013/fut/items/web/199074.json') print r.json ``` Solution ``` #! /usr/bin/env python # -*- coding: utf-8 -*- import json import requests headers = {'Content-Type': 'application/json'} r = requests.get('http://cdn.content.easports.com/fifa/fltOnlineAssets/2013/fut/items/web/199074.json', headers=headers) print r.content #prints {"Item":{"FirstName":"Lacina","LastName":"Traoré","CommonName":null,"Height":"203","DateOfBirth":{"Year":"1990","Month":"8","Day":"20"},"Preferr…

Accepted Answer

For me your site returns `"Traor\u00e9"` (the last character is `é`):

```
r = requests.get(url)
print(json.dumps(json.loads(r.content)['Item']['LastName']))
# -> "Traor\u00e9" -> Traoré
```

`r.json` (`r.text`) produces incorrect content here. Either server or `requests` or both use incorrect encoding that results in `"Traor\u0102\u0160"`. The encoding of JSON text is completely defined by its content therefore it is always possible to decode it whatever headers server sends, from json rfc:

JSON text SHALL be encoded in Unicode. The default encoding is UTF-8.

Since the first two characters of a JSON text will always be ASCII characters [RFC0020], it is possible to determine whether an octet stream is UTF-8, UTF-16 (BE or LE), or UTF-32 (BE or LE) by looking at the pattern of nulls in the first four octets.

```
       00 00 00 xx  UTF-32BE
       00 xx 00 xx  UTF-16BE
       xx 00 00 00  UTF-32LE
       xx 00 xx 00  UTF-16LE
       xx xx xx xx  UTF-8
```

In this case there are no zero bytes at the start of `r.content` so `json.loads` works otherwise you need manually to convert it to a Unicode string if the server sends incorrect character encoding in `Content-Type` header or to workaround `requests` bug

Converting to UTF-8 (again)

Solution

Problem

Related problems