Reading UTF-8 file with codecs in IronPython
csv, encoding, ironpython, python, utf-8
Solution
Looks like it could be a bug in how `next()` handles codecs. Can you please open an issue with the files to reproduce attached?
Problem
I have a .csv file encoded in UTF-8, which contains both latin and cyrillic symbols. ``` ;F1;F2;abcdefg3;F200 ;ABSOLUTE;NOMINAL;NOMINAL;NOMINAL o1;1;USA;Новосибирск;1223 ``` I'm trying to execute following script in IronPython 2.7.1: ``` import codecs f = codecs.open(r"file.csv", "rb", "utf-8") f.next() ``` During the execution of f.next() an exception occurs: ``` Traceback (most recent call last): File "c:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Microsoft\Python Tools for Visual Studio\1.1\visualstudio_py_repl.py", line 492, in run_file_as_main code.Execute(self.exec_mod) File "<string>", line 4, in <module> File "C:\Program Files\IronPython 2.7.1\Lib\codecs.py", line 684, in next return self.reader.next() File "C:\Program Files\IronPython 2.7.1\Lib\codecs.py", line 615, in next line = self.readline() File "C:\Program Files\IronPython 2.7.1\Lib\codecs.py", line 530, in readline data = self.read(readsize, firstline=True) File "C:\Program Files\IronPython 2.7.1\Lib\codecs.py", line 477, in read newchars, decodedbytes = self.decode(data, self.errors) UnicodeEncodeError: ('unknown', '\x00', 0, 1, '') ``` At the same time in CPython 2.7 the script works correctly. Also in the IronPython 2.7.1 following script works fine: ``` import codecs f = codecs.open(r"file.csv", "rb", "utf-8") f.readlines() ``` Does anybody know what may cause such strange behaviour?