Latin-1 and the unicode factory in Python
python, unicode
Solution
Add this at the beginning of the module:
# coding: latin1
Or decode the string to Unicode yourself.
[Edit]
It's been a while since I played with Unicode, but hopefully this example will show how to convert from Latin1 to Unicode:
>>> s = u'ééé'.encode('latin1') # a string you may get from the database
>>> s.decode('latin1')
u'\xe9\xe9\xe9'
[Edit]
Documentation: http://docs.python.org/howto/unicode.html http://docs.python.org/library/codecs.html
Problem
I have a Python 2.6 script that is gagging on special characters, encoded in Latin-1, that I am retrieving from a SQL Server database. I would like to print these characters, but I'm somewhat limited because I am using a library that calls the `unicode` factory, and I don't know how to make Python use a codec other than `ascii`. The script is a simple tool to return lookup data from a database without having to execute the SQL directly in a SQL editor. I use the PrettyTable 0.5 library to display the results. The core of the script is this bit of code. The tuples I get from the cursor contain integer and string data, and no Unicode data. (I'd use `adodbapi` instead of `pyodbc`, which would get me Unicode, but `adodbapi` gives me other problems.) ``` x = pyodbc.connect(cxnstring) r = x.cursor() r.execute(sql) t = PrettyTable(columns) for rec in r: t.add_row(rec) r.close() x.close() t.set_field_align("ID", 'r') t.set_field_align("Name", 'l') print t ``` But the `Name` column can contain characters that fall outside the ASCII range. I'll sometimes get an error message like this, in line 222 of `prettytable.pyc`, when it gets to the `t.add_row` call: ``` UnicodeDecodeError: 'ascii' codec can't decode byte 0xed in position 12: ordinal not in range(128) ``` This is line 222 in `prettytable.py`. It uses `unicode`, which is the source of my problems, and not just in this script, but in other Python scripts that I have written. ``` for i in range(0,len(row)): if len(unicode(row[i])) > self.widths[i]: # This is line 222 self.widths[i] = len(unicode(row[i])) ``` Please tell me what I'm doing wrong here. How can I make `unicode` work without hacking `prettytable.py` or any of the other libraries that I use? Is there even a way to do this? EDIT: The error occurs not at the `print` statement, but at the `t.add_row` call. EDIT: With Bastien Léonard's help, I came up with the following solution. It's not a panacea, but it works. ``` x = pyodbc.connect(cxnstring) r = x.cursor() r.execute(sql) t = PrettyTable(columns) for rec in r: urec = [s.decode('latin-1') if isinstance(s, str) else s for s in rec] t.add_row(urec) r.close() x.close() t.set_field_align("ID", 'r') t.set_field_align("Name", 'l') print t.get_string().encode('latin-1') ``` I ended up having to decode on the way in and encode on the way out. All of this makes me hopeful that everybody ports their libraries to Python 3.x sooner than later!