Python: UnicodeEncodeError when reading from stdin

antlr, python, stdin, unicode

Solution

The problem is, that when reading from stdin, python decodes it using the system default encoding:

>>> import sys
>>> sys.getdefaultencoding()
'ascii'

The input is very likely UTF-8 or Windows-CP-1252, so the program chokes on non-ASCII-characters.

To convert sys.stdin to a stream with the proper decoder, I used:

import codecs
char_stream = codecs.getreader("utf-8")(sys.stdin)

That fixed the problem.

BTW, this is the method ANTLRs FileStream uses to open a file with given filename (instead of a given stream):

    fp = codecs.open(fileName, 'rb', encoding)
    try:
        data = fp.read()
    finally:
        fp.close()

BTW #2: For strings I found

a_string.encode(encoding)

useful.

Problem

When running a Python program that reads from stdin, I get the following error: ``` UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 320: ordinal not in range(128) ``` How can I fix it? Note: The error occurs internal to antlr and the line looks like that: ``` self.strdata = unicode(data) ``` Since I don't want to modify the source code, I'd like to pass in something that is acceptable. The input code looks like that: ``` #!/usr/bin/python import sys import codecs import antlr3 import antlr3.tree from LatexLexer import LatexLexer from LatexParser import LatexParser char_stream = antlr3.ANTLRInputStream(codecs.getreader("utf8")(sys.stdin)) lexer = LatexLexer(char_stream) tokens = antlr3.CommonTokenStream(lexer) parser = LatexParser(tokens) r = parser.document() ```

Original source