Httpclient 4.3 blocking on connection pool
httpclient, java
Solution
I had the same issue and this was the first response. I used Ironluca's comment to solve my problem but felt this needed a full answer.
The quick start guide has a neat example of how to set up and use a basic HttpClient.
// In order to ensure correct deallocation of system resources
// the user MUST call CloseableHttpResponse#close() from a finally clause.
// Please note that if response content is not fully consumed the underlying
// connection cannot be safely re-used and will be shut down and discarded
// by the connection manager.
try {
System.out.println(response1.getStatusLine());
HttpEntity entity1 = response1.getEntity();
// do something useful with the response body
// and ensure it is fully consumed
EntityUtils.consume(entity1);
} finally {
response1.close();
}
Looking back at your question it seems you have the close and consume code. You also have a custom connection pool. It is the same as the default but I guess you have it configured differently.
Problem
when I use httpclient 4.3 as below ``` static { try { SSLContextBuilder builder = new SSLContextBuilder(); builder.loadTrustMaterial(null, new TrustSelfSignedStrategy()); SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(builder.build()); CookieSpecProvider easySpecProvider = new CookieSpecProvider() { public CookieSpec create(HttpContext context) { return new BrowserCompatSpec() { @Override public void validate(Cookie cookie, CookieOrigin origin) throws MalformedCookieException { // Oh, I am easy } }; } }; Registry<CookieSpecProvider> r = RegistryBuilder.<CookieSpecProvider> create() .register(CookieSpecs.BEST_MATCH, new BestMatchSpecFactory()) .register(CookieSpecs.BROWSER_COMPATIBILITY, new BrowserCompatSpecFactory()) .register("easy", easySpecProvider).build(); RequestConfig requestConfig = RequestConfig.custom().setConnectionRequestTimeout(5000) .setSocketTimeout(10000).setConnectTimeout(10000).setCookieSpec("easy").setRedirectsEnabled(false) .build(); PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager(); cm.setMaxTotal(100); cm.setDefaultMaxPerRoute(10); client = HttpClients.custom().setConnectionManager(cm).setDefaultCookieSpecRegistry(r) .setSSLSocketFactory(sslsf).setDefaultRequestConfig(requestConfig).build(); } catch (Exception e) { logger.error("http client init fail!", e); } } public static String execute(HttpRequest httpRequest) { CloseableHttpResponse response = null; HttpGet httpGet = null; HttpEntity httpEntity = null; try { httpGet = new HttpGet(httpRequest.getUrl()); httpGet.setHeader("Connection", "close"); if (httpRequest.isUseGzip()) { httpGet.addHeader("Accept-Encoding", "gzip,deflate,sdch"); } if (!StringUtils.isEmpty(httpRequest.getContentType())) { httpRequest.setContentType(httpRequest.getContentType()); } httpGet.addHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63"); response = client.execute(httpGet); httpEntity = response.getEntity(); byte[] bytes = null; try { bytes = EntityUtils.toByteArray(httpEntity); } catch (Exception e) { return null; } if (response.getStatusLine().getStatusCode() != 200) { logger.warn("error! StatusCode: " + response.getStatusLine().getStatusCode() + ", url: " + httpRequest.getUrl()); return null; } @SuppressWarnings("deprecation") String charset = EntityUtils.getContentCharSet(httpEntity); if (StringUtils.isEmpty(charset)) { Matcher match = charsetPatterm.matcher(new String(bytes)); if (match.find()) { charset = match.group(1); } } if (!StringUtils.isEmpty(charset)) { String strUtf8 = new String(new String(bytes, charset).getBytes(), GlobalConfig.ENCODING); return StringEscapeUtils.unescapeHtml4(strUtf8); } } catch (Exception e) { logger.error("error! url [" + httpRequest.getUrl() + "]", e); } finally { try { if (httpEntity != null) { EntityUtils.consume(httpEntity); } if (response != null) { response.close(); } if (httpGet != null) { httpGet.abort(); } } catch (Exception e) { // ignore } } return null; } ``` thread will block.. jstack show like this. I just use it to crawl some website. It happens when statusCode is 404. Using Java Apache PoolingClientConnectionManager leaks the Memory,How to solve it? my problem is similar to this. ``` "pool-1-thread-10" prio=10 tid=0x00007f7168003000 nid=0x3e4d waiting on condition [0x00007f717c398000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000e69d7350> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:133) at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:282) at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64) at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:177) at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:170) at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:102) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.jav a:244) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:231) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:173) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106) ``` how to solve?