python twisted agent timeout

Question

I'm quite new to twisted and I'm trying to make an asynchronous client that fetches some urls and saves the result into a distinct file for each url. When I run the program with a limited amount of servers let's say 10, the reactor loop correctly ends and the program terminates. But when I run the program with for example the Alexa top 2500, the program starts to fetch urls but then does not terminate. I've set the timeout but it does not work, I believe that there must be some open socket that does not trigger any callback either for error or success. My aim is once the program has fetched the pages or the per-connection timeout has expired the program has to terminate and close all the file descriptors active. I'm Sorry but the code indentation is not kept while copy and pasting, now I've checked and it's fixed. The code is the bare minimum to give an example, note that with my problem is that the reactor is not stopping when I start the program with a huge number of sites to crawl.…

Accepted Answer

There are at least two issues with your timeout code.

First, the only timeout you set is `_connectTimeout`, and you set it on a `Deferred` returned from `Agent.request`. This is a meaningless attribute, and nothing in the `Agent` implementation nor any part of Twisted will respect it. I think you meant to set this attribute on the `Agent` instance instead, where it would have had some affect. However, it is a private attribute not meant for you to interact with directly. Instead, you should pass `connectTimeout=10` to the `Agent` initializer.

Second, this timeout only affects the TCP connection setup timeout. Setting it to `10` means that if a TCP connection to the HTTP server for a particular URL cannot be established in fewer than 10 seconds, the request attempt will be failed with a timeout error. If the connection is successfully established in less than 10 seconds, though, the timeout has no further meaning. If the server takes 10 hours to send you a response, `Agent` will sit there and wait for 10 hours. You need an additional timeout, a whole-request timeout.

This is something to implement separately using `reactor.callLater` and possibly `Deferred.cancel`. For example,

```
...
d = agent.request(...)
timeoutCall = reactor.callLater(60, d.cancel)
def completed(passthrough):
    if timeoutCall.active():
        timeoutCall.cancel()
    return passthrough
d.addBoth(completed)
...
```

python twisted agent timeout

Solution

Problem