Asynchronous Requests with Python requests
asynchronous, httprequest, python, python-requests
Solution
Note
The below answer is not applicable to requests v0.13.0+. The asynchronous functionality was moved to grequests after this question was written. However, you could just replace `requests` with `grequests` below and it should work.
I've left this answer as is to reflect the original question which was about using requests < v0.13.0.
To do multiple tasks with `async.map` asynchronously you have to:
- Define a function for what you want to do with each object (your task)
- Add that function as an event hook in your request
- Call `async.map` on a list of all the requests / actions
Example:
from requests import async
# If using requests > v0.13.0, use
# from grequests import async
urls = [
'http://python-requests.org',
'http://httpbin.org',
'http://python-guide.org',
'http://kennethreitz.com'
]
# A simple task to do to each response object
def do_something(response):
print response.url
# A list to hold our things to do via async
async_list = []
for u in urls:
# The "hooks = {..." part is where you define what you want to do
#
# Note the lack of parentheses following do_something, this is
# because the response will be used as the first argument automatically
action_item = async.get(u, hooks = {'response' : do_something})
# Add the task to our list of things to do via async
async_list.append(action_item)
# Do our list of things to do via async
async.map(async_list)
Problem
I tried the sample provided within the documentation of the requests library for python. With `async.map(rs)`, I get the response codes, but I want to get the content of each page requested. This, for example, does not work: ``` out = async.map(rs) print out[0].content ```