Error codes returned by urllib/urllib2 and the actual page

error-handling, python

Solution

urllib2 raises a `HTTPError` when HTTP errors happen. You can get to the response code using `code` on the exception object. You can get the response data using `read()`:

>>> req = urllib2.Request('http://www.python.org/fish.html')
>>> try:
>>>     urllib2.urlopen(req)
>>> except urllib2.HTTPError, e:
>>>     print e.code
>>>     print e.read()
>>>
404
<actual data response will be here>

Problem

the normal behavior of urllib/urllib2 is if an error code is sent in the header of the response (i.e 404) an Exception is raised. How do you look for specific errors i.e (40x, or 50x) based on the different errors, do different things. Also, how do you read the actual data being returned HTML/JSON etc (The data usually has error details which is different to the HTML error code)

Original source