How to Speed Up Python's urllib2 when doing multiple requests
http, python, urllib2
Solution
If you switch to httplib, you will have finer control over the underlying connection.
For example:
import httplib
conn = httplib.HTTPConnection(url)
conn.request('GET', '/foo')
r1 = conn.getresponse()
r1.read()
conn.request('GET', '/bar')
r2 = conn.getresponse()
r2.read()
conn.close()
This would send 2 HTTP GETs on the same underlying TCP connection.
Problem
I am making several http requests to a particular host using python's urllib2 library. Each time a request is made a new tcp and http connection is created which takes a noticeable amount of time. Is there any way to keep the tcp/http connection alive using urllib2?