UnicodeDammit: detwingle crashes on a website
beautifulsoup, python, unicode
Solution
This website is not a special case in terms of character encoding at all, it's perfectly valid utf-8 with even the http header set correctly. It then follows that your code would have crashed on any website encoded in utf-8 with code points beyond ASCII.
It is also evident from the documentation, that `UnicodeDammit.detwingle` takes an unicode string. You are passing it `html_blob`, and the variable naming suggests that it's not a decoded unicode string. (Misunderstanding)
To handle any website encoding is not trivial in the case the http header or markup lies about the encoding or is not included at all. You need to perform various heuristics and even then you won't get it right. But this website is sending the charset header correctly and has been encoded correctly in that charset.
Interesting trivia. The only beyond ASCII text in the website are these javascript comments (after being decoded as utf-8):
image = new Array(4); //¶¨ÒåimageΪͼƬÊýÁ¿µÄÊý×é
image[0] = 'sample_BG_image01.png' //±³¾°Í¼ÏóµÄ·¾¶
If you then encode those to ISO-8859-1, and decode the result as GB2312, you get:
image = new Array(4); //定义image为图片数量的数组
image[0] = 'sample_BG_image01.png' //背景图象的路径
Which google chinese -> english, translates to:
image = new Array(4); //Defined image of the array of the number of images
image[0] = 'sample_BG_image01.png' //The path of the background image
Problem
I’m scraping websites and use BeautifulSoup4 to parse them. As the websits can have really random char sets, I use UnicodeDammit.detwingle to ensure that I feed proper data to BeautifulSoup. It worked fine... until it crashed. One website causes the code to break. The code to build "soup" looks like this: ``` u = bs.UnicodeDammit.detwingle( html_blob ) <--- here it crashes u = bs.UnicodeDammit( u.decode('utf-8'), smart_quotes_to='html', is_html = True ) u = u.unicode_markup soup = bs.BeautifulSoup( u ) ``` And the error (standard Python-Unicode hell duo) ``` File ".../something.py", line 92, in load_bs_from_html_blob u = bs.UnicodeDammit.detwingle( html_blob ) File ".../beautifulsoup4-4.1.3-py2.7.egg/bs4/dammit.py", line 802, in detwingle return b''.join(byte_chunks) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 0: ordinal not in range(128) ``` The offending website is this one Question: How to make a proper and bulletproof website source decoding?