Unicode error when outputting python script output to file
beautifulsoup, python, unicode
Solution
You can use the codecs module to write unicode data to the file
import codecs
file = codecs.open("out.txt", "w", "utf-8")
file.write(something)
'print' outputs to the standart output and if your console doesn't support utf-8 it can cause such error even if you pipe stdout to a file.
Problem
This is the code: ``` print '"' + title.decode('utf-8', errors='ignore') + '",' \ ' "' + title.decode('utf-8', errors='ignore') + '", ' \ '"' + desc.decode('utf-8', errors='ignore') + '")' ``` title and desc are returned by Beautiful Soup 3 (p[0].text and p[0].prettify) and as far as I can figure out from BeautifulSoup3 documentation are UTF-8 encoded. If I run ``` python.exe script.py > out.txt ``` I get following error: ``` Traceback (most recent call last): File "script.py", line 70, in <module> '"' + desc.decode('utf-8', errors='ignore') + '")' UnicodeEncodeError: 'ascii' codec can't encode character u'\xf8' in position 264 : ordinal not in range(128) ``` However if I run ``` python.exe script.py ``` I get no error. It happens only if output file is specified. How to get good UTF-8 data in the output file?