Python and BeautifulSoup encoding issues
beautifulsoup, python, unicode, utf-8
Solution
could you try:
r = urllib.urlopen('http://www.elnorte.ec/')
x = BeautifulSoup.BeautifulSoup(r.read)
r.close()
print x.prettify('latin-1')
I get the correct output. Oh, in this special case you could also `x.__str__(encoding='latin1')`.
I guess this is because the content is in ISO-8859-1(5) and the meta http-equiv content-type incorrectly says "UTF-8".
Could you confirm?
Problem
I'm writing a crawler with Python using BeautifulSoup, and everything was going swimmingly till I ran into this site: http://www.elnorte.ec/ I'm getting the contents with the requests library: ``` r = requests.get('http://www.elnorte.ec/') content = r.content ``` If I do a print of the content variable at that point, all the spanish special characters seem to be working fine. However, once I try to feed the content variable to BeautifulSoup it all gets messed up: ``` soup = BeautifulSoup(content) print(soup) ... <a class="blogCalendarToday" href="/component/blog_calendar/?year=2011&month=08&day=27&modid=203" title="1009 artÃculos en este dÃa"> ... ``` It's apparently garbling up all the spanish special characters (accents and whatnot). I've tried doing content.decode('utf-8'), content.decode('latin-1'), also tried messing around with the fromEncoding parameter to BeautifulSoup, setting it to fromEncoding='utf-8' and fromEncoding='latin-1', but still no dice. Any pointers would be much appreciated.