Random "[Errno -2] Name or service not known" errors

django, networking, python, urllib

Solution

if you use a bad proxy or there are network problems you can try this:

for url in urllist:
    retry = 0
    while True: # retry request
        try:
            req = urllib.urlopen(url)
            resp_data = req.read() # in call read() network still processing
        except Exception as e: # TODO need more detailed handling
            if retry > 3: # 3 this is serious problem. exit
                raise e
            retry += 1 # retry
        else:
            data = json.loads()
            req.close() # not needed
            break

Problem

I am populating a local database using a third party service. I have a list of urls (around 500). I am calling each url in a loop, and updating my database with the returned data. The code flow looks like this: ``` for url in urllist: req = urllib.urlopen(url) data = json.loads(req.read()) req.close() #update the db using data here ``` Whenever I run this piece of code, the script fails at random points with the error message "Name or service not known". This doesn't have anything with the urls because the script fails at random points (i.e. at 50th iteration in one run, and at 60th iteration in another) What could be the reason for this?

Original source