How to catch 404 error in urllib.urlretrieve
http, python, url, urllib
Solution
Check out `urllib.urlretrieve`'s complete code:
def urlretrieve(url, filename=None, reporthook=None, data=None):
global _urlopener
if not _urlopener:
_urlopener = FancyURLopener()
return _urlopener.retrieve(url, filename, reporthook, data)
In other words, you can use urllib.FancyURLopener (it's part of the public urllib API). You can override `http_error_default` to detect 404s:
class MyURLopener(urllib.FancyURLopener):
def http_error_default(self, url, fp, errcode, errmsg, headers):
# handle errors the way you'd like to
fn, h = MyURLopener().retrieve(url, reporthook=my_report_hook)
Problem
Background: I am using `urllib.urlretrieve`, as opposed to any other function in the `urllib*` modules, because of the hook function support (see `reporthook` below) .. which is used to display a textual progress bar. This is Python >=2.6. ``` >>> urllib.urlretrieve(url[, filename[, reporthook[, data]]]) ``` However, `urlretrieve` is so dumb that it leaves no way to detect the status of the HTTP request (eg: was it 404 or 200?). ``` >>> fn, h = urllib.urlretrieve('http://google.com/foo/bar') >>> h.items() [('date', 'Thu, 20 Aug 2009 20:07:40 GMT'), ('expires', '-1'), ('content-type', 'text/html; charset=ISO-8859-1'), ('server', 'gws'), ('cache-control', 'private, max-age=0')] >>> h.status '' >>> ``` What is the best known way to download a remote HTTP file with hook-like support (to show progress bar) and a decent HTTP error handling?