What is the fastest way to send thousands of post requests with Python?

http, post, python

Solution

The way you've written your code, it waits for the response to one request before sending the next. (On top of that, it may not reuse the HTTP connections, meaning you have to deal with the socket creation/shutdown overhead for each request. Then again, depending on what you're testing, there's a good chance that actually makes it a better test.)

The simplest way to make multiple requests at the same time is to use threads. And the easiest way to do that is with `concurrent.futures` (or `futures` from PyPI, if you're using 2.x or 3.1):

with concurrent.futures.ThreadPoolExecutor(max_workers=10) as pool:
    results = pool.map(post, words)
    concurrent.futures.wait(results)

If you prefer, you can write your own threads and just give each thread 1/10th of `words` and have it loop over calling `post`:

def posts(words):
    for word in words:
        post(word)

groupsize = len(words)/10
t = [threading.Thread(target=posts, args=[words[i*groupsize:(i+1)*groupsize]]
     for i in range(10)]
for thread in t:
    thread.start()
for thread in t:
    thread.join()

Either way, obviously I just pulled that number 10 out of thin air (because it's a little more than the max simultaneous connections most browsers or web service clients will allow you to create), but you'll want to do some performance testing to find the best value.

If it turns out that the best value is huge, like 500 or something, you may be running into the limits of what you can do with threading. In that case, you should consider using greenlets. The simplest way to do this is with `gevent`—and the simplest to do that is to rewrite your code to use `grequests` instead of `urllib2`.

Meanwhile, if the actual reads are wasting time, and you don't actually need the responses, and they're reasonably big, and you're not trying to test the server's ability to send real responses, you may want to close the socket as soon as you know you're going to get the right data. You can do this with `urllib2` by writing your own handlers, but that sounds like a lot of work. I think it would actually be simpler, in this case, to just drop down to the level of sockets. First, record the request that gets sent for each POST, and the expected 200 line that you get back when things work. Then do something like this:

with closing(socket.socket()) as c:
    c.connect(('127.0.0.1', 8000))
    c.send(REQUEST_STRING_FORMAT.format([word]))
    with c.makefile() as f:
        response = f.readline()
        if response != RESPONSE_200_STRING:
            response += f.read()
            with open('error.html','w') as k:
                k.write(response)

Problem

I want to make thousands of POST requests in the fastest possible way. How can I do this with Python? So far I just have a for loop which calls the following function many times: ``` def post(word): data = json.dumps({"word":word}) url = "http://127.0.0.1:8000/updateword" req = urllib2.Request(url, data, {'Content-Type': 'application/octet-stream'}) response=None try: f = urllib2.urlopen(req) response = f.read() f.close() except urllib2.HTTPError, error: k= open('error.html','w') k.write(error.read()) k.close() return response ``` Is there a better way to do this?

Original source