speeding up urlib.urlretrieve

python, urllib

Solution

my code above was very naive as I did not take advantage of multi-threading. It obviously takes for url requests to be responded but there is no reason why the computer cannot make further requests while the proxy server responds.

Doing the following adjustments, you can improve efficiency by 10x - and there are further ways for improving efficiency, with packages such as scrapy.

To add multi-threading, do something like the following, using the multiprocessing package:

1) encapsulate the url retrieving in a function:

import import urllib.request

def geturl(link,i):
try:
    urllib.request.urlretrieve(link, str(i)+".jpg")
except:
    pass

2) then create a collection with all urls as well as names you want for the downloaded pictures:

urls = [url1,url2,url3,urln]
names = [i for i in range(0,len(urls))]

3)Import the Pool class from the multiprocessing package and create an object using such class (obviously you would include all imports in the first line of your code in a real program):

from multiprocessing.dummy import Pool as ThreadPool
pool = ThreadPool(100)

then use the pool.starmap() method and pass the function and the arguments of the function.

results = pool.starmap(geturl, zip(links, d))

note: pool.starmap() works only in Python 3

Problem

I am downloading pictures from the internet, and as it turns out, I need to download lots of pictures. I am using a version of the following code fragment (actually looping through the links I intend to download and downloading the pictures : ``` import urllib urllib.urlretrieve(link, filename) ``` I am downloading roughly 1000 pictures every 15 minutes, which is awfully slow based on the number of pictures I need to download. For efficiency, I set a timeout every 5 seconds (still many downloads last much longer): ``` import socket socket.setdefaulttimeout(5) ``` Besides running a job on a computer cluster to parallelize downloads, is there a way to make the picture download faster / more efficient?

Original source