Decoding HTML entities with Python

beautifulsoup, character-encoding, content-type, python, unicode

Solution

Try this:

import re

def _callback(matches):
    id = matches.group(1)
    try:
        return unichr(int(id))
    except:
        return id

def decode_unicode_references(data):
    return re.sub("&#(\d+)(;|(?=\s))", _callback, data)

data = "U.S. Adviser’s Blunt Memo on Iraq: Time ‘to Go Home’"
print decode_unicode_references(data)

Problem

I'm trying to decode HTML entries from here NYTimes.com and I cannot figure out what I am doing wrong. Take for example: ``` "U.S. Adviser’s Blunt Memo on Iraq: Time ‘to Go Home’" ``` I've tried BeautifulSoup, decode('iso-8859-1'), and django.utils.encoding's smart_str without any success.

Original source