How to handle the encode in lxml to parse html-string properly?

Question

I have a xml file. please download it and save it as `blog.xml`. It is the list of my files in Google-blogger, i write some codes to parse it ,there is a something wring with lxml . code1: ``` from stripogram import html2text import feedparser d = feedparser.parse('blog.xml') for num,entry in enumerate(d.entries): string=entry.content[0]['value'].encode("utf-8") print html2text(string) ``` It get a right result with code1. code2: ``` import lxml.html import feedparser d = feedparser.parse('blog.xml') for num,entry in enumerate(d.entries): string=entry.content[0]['value'] myhtml=lxml.html.document_fromstring(string) print myhtml.text_content() ``` It get a wrong output with code2. ``` Traceback (most recent call last): File "", line 3, in File "/usr/lib/python2.7/dist-packages/lxml/html/__init__.py", line 532, in document_fromstring value = etree.fromstring(html, parser, **kw) File "lxml.etree.pyx", line 2754, in lxml.etree.fromstring (src/lxml/lxml.etree.c:54631) File…

Accepted Answer

There is a bug in lxml. Check output of this code:

```
import lxml.html
import feedparser

def test():
    try:
        lxml.html.document_fromstring('')
    except Exception as e:
        print e

d = feedparser.parse('blog.xml')
e = d.entries[0].content[0]['value'].encode('utf-8')

test() # XMLSyntaxError: None

lxml.html.document_fromstring(e)
test() # XMLSyntaxError: line 1407: Tag b:include invalid
```

So the error is confusing, the real reason why your parsing fails is that you pass empty strings to document_fromstring.

Try this code:

```
import lxml.html
import feedparser
d = feedparser.parse('blog.xml')
for num,entry in enumerate(d.entries):
    string=entry.content[0]['value'].encode("utf-8") 
    if not string:
        continue
    myhtml=lxml.html.document_fromstring(string)
    print  myhtml.text_content()
```

How to handle the encode in lxml to parse html-string properly?

Solution

Problem