Apache's HttpClient stops working when MaxPoolSize in PoolingHttpClientConnectionManager is too small

Question

I have a `SwingWorker` that gets a list of URLs (can be several hundred). In the `doInBackground()` method it loops through these URLs, creates each time a `CloseableHttpClient`, managed by a `PoolingHttpClientConnectionManager`, and a `HttpGet`. The `client` will then execute the `httpGet` and write the content (image) to a file (if possible! Not all URLs are valid and some return a `404`). This works for approximately 100 requests just until the `maxTotal` (or the `defaultMaxPerRoute`) of the connectionManager is reached. Then everything stops, the client's execution stops and no Exception gets thrown. So I figured, I'd set the `maxTotal` and the `defaultMaxPerRoute` to `1000` and check it out. It works when I try downloading 1500 images, but it just feels so wrong! I want to reuse the clients instead of having 1000 of them in a pool. Socket- or ConnectionTimeouts didn't work, Debugging does not tell me what is happening and creating new `HttpClients` without the `PoolingHttpClientC…

Accepted Answer

You may be leaking connections due to the fact that you only properly dispose of them when you have OK responses (in the range 200 to 300) that you can dump to your file output stream properly.

Long story short : always dispose of the entity

Disposing of the entity (the "content" of the resposne) is the only way to free the connexion object. To that end, you can call `EntityUtils.consume(responseEntity)`) and/or close the response object (call `response.close()`) and/or the close or read up to its end the entity's stream (call `stream.close()`) in a `finally` clause for each request/response cycle.

Your code actually consumes the response's stream, but only for certain cases (mainly: when the request succeeds), so that is a proper use of the API, but not otherwise (e.g. 404 responses), that you never consume nor release.

You can also use the `ResponseHandler` API variant for that, and it will hande ALL of this for you - i'd advise you to use it.

If you do not always do that, the connection will be considered active by HTTPClient, and it will never be released, nor reused. At some point you will end up with either an empty connection pool, or a system error (e.g. too many files open).

Please note that it's not only for 200 OK response, it's for every call (even a 404 usually has a response body, if you do not consume that body, then the connection will not be freed).

Hints from the user guide

I refer you to the "fundamentals" part of the user guide : https://hc.apache.org/httpcomponents-client-ga/tutorial/html/fundamentals.html

Here is an example of request execution process in its simplest form:

```
CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpget = new HttpGet("http://localhost/");
CloseableHttpResponse response = httpclient.execute(httpget);
try {
    <...>
} finally {
    response.close();
}
```

The very important part is the close() in the finally close.

1.1.5. Ensuring release of low level resources

In order to ensure proper release of system resources one must close either the content stream associated with the entity or the response itself

And :

1.1.6. Consuming entity content

The recommended way to consume the content of an entity is by using its HttpEntity#getContent() or HttpEntity#writeTo(OutputStream) methods. HttpClient also comes with the EntityUtils class, which exposes several static methods to more easily read the content or information from an entity. Instead of reading the java.io.InputStream directly, one can retrieve the whole content body in a string / byte array by using the methods from this class. However, the use of EntityUtils is strongly discouraged unless the response entities originate from a trusted HTTP server and are known to be of limited length.

About the response handler variant :

1.1.8. Response handlers

The simplest and the most convenient way to handle responses is by using the ResponseHandler interface, which includes the handleResponse(HttpResponse response) method. This method completely relieves the user from having to worry about connection management.

Hints from stack overflow

You can also read this : Why did the author use EntityUtils.consume(httpEntity);?

Sidenote

The point of having multithreaded connection managers is that you can have ONE (and only one) client, and share it for your whole application. You do not need to build one client per request and have them share a single connection manager. Although you can. Having a single client could help if you use caching, cookies, authentication, ... thant can not work when creating a new client for each request.

1.2.1. HttpClient thread safety

HttpClient implementations are expected to be thread safe. It is recommended that the same instance of this class is reused for multiple request executions.

Apache's HttpClient stops working when MaxPoolSize in PoolingHttpClientConnectionManager is too small

Solution

Problem

Related problems