Python ElementTree won't convert non-breaking spaces when using UTF-8 for output

elementtree, encoding, python, xml

Solution

0xA0 is a latin1 character, not a unicode character and the value of p.text in the loop is a str and not unicode, that means that in order to encode it in utf-8 it must first be converted by Python implicitly into a unicode string (i.e. using decode). When it is doing this it assumes ascii since it wasn't told anything else. 0xa0 is not a valid ascii character, but it is a valid latin1 character.

The reason you have latin1 characters instead of unicode characters is because entitydefs is a mapping of names to latin1 encode strings. You need the unicode code point which you can get from htmlentitydef.name2codepoint

The version below should fix it for you:

import sys
from cStringIO  import StringIO
from xml.etree  import ElementTree as ET
from htmlentitydefs import name2codepoint

source = StringIO("""<html>
<body>
<p>Less than &lt;</p>
<p>Non-breaking space &nbsp;</p>
</body>
</html>""")

parser = ET.XMLParser()
parser.parser.UseForeignDTD(True)
parser.entity.update((x, unichr(i)) for x, i in name2codepoint.iteritems())
etree = ET.ElementTree()

tree = etree.parse(source, parser=parser)
for p in tree.findall('.//p'):
    print ET.tostring(p, encoding='UTF-8')

Problem

I'm trying to parse, manipulate, and output HTML using Python's ElementTree: ``` import sys from cStringIO import StringIO from xml.etree import ElementTree as ET from htmlentitydefs import entitydefs source = StringIO("""<html> <body> <p>Less than &lt;</p> <p>Non-breaking space &nbsp;</p> </body> </html>""") parser = ET.XMLParser() parser.parser.UseForeignDTD(True) parser.entity.update(entitydefs) etree = ET.ElementTree() tree = etree.parse(source, parser=parser) for p in tree.findall('.//p'): print ET.tostring(p, encoding='UTF-8') ``` When I run this using Python 2.7 on Mac OS X 10.6, I get: ``` <p>Less than &lt;</p> Traceback (most recent call last): File "bar.py", line 20, in <module> print ET.tostring(p, encoding='utf-8') File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1120, in tostring ElementTree(element).write(file, encoding, method=method) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 815, in write serialize(write, self._root, encoding, qnames, namespaces) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 931, in _serialize_xml write(_escape_cdata(text, encoding)) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1067, in _escape_cdata return text.encode(encoding, "xmlcharrefreplace") UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position 19: ordinal not in range(128) ``` I thought that specifying "encoding='UTF-8'" would take care of the non-breaking space character, but apparently it doesn't. What should I do instead?

Original source