How to encode/decode this file in Python?

Question

I am planning to make a little Python game that will randomly print keys (English) out of a dictionary and the user has to input the value (in German). If the value is correct, it prints 'correct' and continue. If the value is wrong, it prints 'wrong' and breaks. I thought this would be an easy task but I got stuck on the way. My problem is I do not know how to print the German characters. Let's say I have a file 'dictionary.txt' with this text: ``` cat:Katze dog:Hund exercise:Übung solve:lösen door:Tür cheese:Käse ``` And I have this code just to test how the output looks like: ``` # -*- coding: UTF-8 -*- words = {} # empty dictionary with open('dictionary.txt') as my_file: for line in my_file.readlines(): if len(line.strip())>0: # ignoring blank lines elem = line.split(':') # split on ":" words[elem[0]] = elem[1].strip() # appending elements to dictionary print words ``` Obviously the result of the print is not as expected: ``` {'cheese': 'K\xc3\xa4se', 'door': 'T\xc3\xbcr', 'dog':…

Accepted Answer

You are looking at byte string values, printed as `repr()` results because they are contained in a dictionary. String representations can be re-used as Python string literals and non-printable and non-ASCII characters are shown using string escape sequences. Container values are always represented with `repr()` to ease debugging.

Thus, the string 'K\xc3\xa4se' contains two non-ASCII bytes with hex values C3 and A4, a UTF-8 combo for the U+00E4 codepoint.

You should decode the values to `unicode` objects:

```
with open('dictionary.txt') as my_file:
    for line in my_file:   # just loop over the file
        if line.strip(): # ignoring blank lines
            key, value = line.decode('utf8').strip().split(':')
            words[key] = value
```

or better still, use `codecs.open()` to decode the file as you read it:

```
import codecs

with codecs.open('dictionary.txt', 'r', 'utf8') as my_file:
    for line in my_file:
        if line.strip(): # ignoring blank lines
            key, value = line.strip().split(':')
            words[key] = value
```

Printing the resulting dictionary will still use `repr()` results for the contents, so now you'll see `u'cheese': u'K\xe4se'` instead, because `\xe4` is the escape code for Unicode point 00E4, the `ä` character. Print individual words if you want the actual characters to be written to the terminal:

```
print words['cheese']
```

But now you can compare these values with other data that you decoded, provided you know their correct encoding, and manipulate them and encode them again to whatever target codec you needed to use. `print` will do this automatically, for example, when printing unicode values to your terminal.

You may want to read up on Unicode and Python:

The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) by Joel Spolsky

The Python Unicode HOWTO

Pragmatic Unicode by Ned Batchelder

How to encode/decode this file in Python?

Solution

Problem