Default encoding of exception messages

encoding, exception, python, python-2.x

Solution

e[0] isn't encoded with latin-1; it just so happens that the byte \xbd, when decoded as latin-1, is the character U+00BD.

The conversion occurs in `Objects/floatobject.c`.

First, the unicode string must be converted to a byte string. This is performed using `PyUnicode_EncodeDecimal()`:

if (PyUnicode_EncodeDecimal(PyUnicode_AS_UNICODE(v),
                            PyUnicode_GET_SIZE(v),
                            s_buffer,
                            NULL))
        return NULL;

which is implemented in `unicodeobject.c`. It doesn't perform any sort of character set conversion, it just writes bytes with values equal to the unicode ordinals of the string. In this case, U+00BD -> 0xBD.

The statement formatting the error is:

PyOS_snprintf(buffer, sizeof(buffer),
              "invalid literal for float(): %.200s", s);

where `s` contains the byte string created earlier. `PyOS_snprintf()` writes a byte string, and `s` is a byte string, so it just includes it directly.

Problem

The following code examines the behaviour of the `float()` method when fed a non-ascii symbol: ``` import sys try: float(u'\xbd') except ValueError as e: print sys.getdefaultencoding() # in my system, this is 'ascii' print e[0].decode('latin-1') # u'invalid literal for float(): ' followed by the 1/2 (one half) character print unicode(e[0]) # raises "UnicodeDecodeError: 'ascii' codec can't decode byte 0xbd in position 29: ordinal not in range(128)" ``` My question: why is the error message `e[0]` encoded in Latin-1? The default encoding is Ascii, and this seems to be what `unicode()` expects. Platform is Ubuntu 9.04, Python 2.6.2

Original source