Shulou(Shulou.com)06/01 Report--

This article introduces the knowledge of "what is HttpClient connection pool and retry mechanism". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

A brief introduction to HttpClient

HttpClient is a subproject under Apache Jakarta Common, which can be used to provide efficient, up-to-date, feature-rich client programming toolkits that support the HTTP protocol, based on the standard Java language.

Function introduction

Support for HTTP and HTTPS protocols

Realize the method of HTTP, GET,POST,PUT,DELETE and so on.

The connection manager supports multithreaded applications.

You can set the connection timeout

Usage

Using HttpClient to send a request, receiving a response can be divided into the following steps:

Create a HttpClient object

Create an instance of the request method and specify URL

Send request parameters, GET request and POST request send parameters in different ways

Call the execute method of the HttpClient object and return the HttpResponse object

Call HttpResponse's getAllHeaders (), getHeaders (String name) and other methods to get the server's response header; call HttpResponse's getEntity () method to get the HttpEntity object, which wraps the server's response content

Connection released. Whether successful or not, the connection must be released

II. HttpClientUtil2.1 HttpClient version

The version used by the author is 4.5.5. Because it is a maven project, it is necessary to introduce the corresponding coordinates in the pom file.

The tool classes used in the org.apache.httpcomponents httpclient 4.5.52.2 project are as follows: package cn.htjc.customer.util;import lombok.extern.slf4j.Slf4j;import org.apache.http.HttpResponse;import org.apache.http.NameValuePair;import org.apache.http.client.ServiceUnavailableRetryStrategy;import org.apache.http.client.config.RequestConfig;import org.apache.http.client.entity.UrlEncodedFormEntity Import org.apache.http.client.methods.CloseableHttpResponse;import org.apache.http.client.methods.HttpGet;import org.apache.http.client.methods.HttpPost;import org.apache.http.client.utils.URIBuilder;import org.apache.http.config.Registry;import org.apache.http.config.RegistryBuilder;import org.apache.http.conn.socket.ConnectionSocketFactory;import org.apache.http.conn.socket.PlainConnectionSocketFactory;import org.apache.http.conn.ssl.SSLConnectionSocketFactory;import org.apache.http.conn.ssl.TrustSelfSignedStrategy Import org.apache.http.entity.ContentType;import org.apache.http.entity.StringEntity;import org.apache.http.impl.client.CloseableHttpClient;import org.apache.http.impl.client.DefaultHttpRequestRetryHandler;import org.apache.http.impl.client.HttpClients;import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;import org.apache.http.message.BasicNameValuePair;import org.apache.http.protocol.HttpContext;import org.apache.http.ssl.SSLContextBuilder;import org.apache.http.util.EntityUtils;import java.io.IOException;import java.net.SocketTimeoutException Import java.net.URI;import java.util.ArrayList;import java.util.List;import java.util.Map;@Slf4jpublic class HttpClientUtil {/ / utf-8 character encoding private static final String CHARSET_UTF_8 = "utf-8"; / / HTTP content type. Equivalent to a form form, submit data private static final String CONTENT_TYPE_FORM_URL = "application/x-www-form-urlencoded"; / / connection manager private static PoolingHttpClientConnectionManager pool; / / request configuration private static RequestConfig requestConfig; static {try {log.info ("initial custom HttpClient. Start "); SSLContextBuilder builder = new SSLContextBuilder (); builder.loadTrustMaterial (null, new TrustSelfSignedStrategy ()); SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory (builder.build ()) / / configuration supports both HTTP and HTPPS Registry socketFactoryRegistry = RegistryBuilder.create () .register ("http", PlainConnectionSocketFactory.getSocketFactory ()) .register ("https", sslsf) .build (); / / initializes the connection manager pool = new PoolingHttpClientConnectionManager (socketFactoryRegistry) / / set the maximum number of connections for the connection pool pool.setMaxTotal (200); / / set the default number of connections per route pool.setDefaultMaxPerRoute (20); / / initialize the requestConfig according to the default timeout limit / / the timeout int socketTimeout for clients to read data from the server = 1000 / / timeout int connectTimeout for establishing connections between client and server = 10000; / / timeout int connectionRequestTimeout for obtaining connections from connection pool = 10000; / / setting request timeout requestConfig = RequestConfig.custom (). SetConnectionRequestTimeout (connectionRequestTimeout) .setSocketTimeout (socketTimeout). SetConnectTimeout (connectTimeout). Build () Log.info ("initial custom HttpClient." End ");} catch (Exception e) {log.error (" initial custom HttpClient. Private HttpClientUtil () {} private static CloseableHttpClient getHttpClient () {/ / when the status code is 503, the policy takes effect ServiceUnavailableRetryStrategy serviceUnavailableRetryStrategy = new ServiceUnavailableRetryStrategy () {@ Override public boolean retryRequest (HttpResponse httpResponse, int I, HttpContext httpContext) {if (I)

< 3) { log.info("ServiceUnavailableRetryStrategy========================"+i); return true; } return false; } @Override public long getRetryInterval() { return 2000L; } }; CloseableHttpClient httpClient = HttpClients.custom() // 设置连接池管理 .setConnectionManager(pool) // 设置请求配置 .setDefaultRequestConfig(requestConfig) // 设置重试次数 .setRetryHandler(new DefaultHttpRequestRetryHandler()) .setServiceUnavailableRetryStrategy(serviceUnavailableRetryStrategy) .build(); return httpClient; } public static String doGet(String url, Map param) { // 创建Httpclient对象 CloseableHttpClient httpClient = getHttpClient(); String resultString = ""; CloseableHttpResponse response = null; try { // 创建uri URIBuilder builder = new URIBuilder(url); if (param != null) { for (String key : param.keySet()) { builder.addParameter(key, param.get(key)); } } URI uri = builder.build(); // 创建http GET请求 HttpGet httpGet = new HttpGet(uri); // 执行请求 response = httpClient.execute(httpGet); // 判断返回状态是否为200 if (response.getStatusLine().getStatusCode() == 200) { resultString = EntityUtils.toString(response.getEntity(), CHARSET_UTF_8); } } catch (Exception e) { e.printStackTrace(); } finally { try { if (response != null) { response.close(); } } catch (IOException e) { e.printStackTrace(); } } return resultString; } public static String doGet(String url) { return doGet(url, null); } public static String doPost(String url, Map param) { // 创建Httpclient对象 CloseableHttpClient httpClient = getHttpClient(); CloseableHttpResponse response = null; String resultString = ""; try { // 创建Http Post请求 HttpPost httpPost = new HttpPost(url); // 创建参数列表 if (param != null) { List paramList = new ArrayList(); for (String key : param.keySet()) { paramList.add(new BasicNameValuePair(key, param.get(key))); } // 模拟表单 UrlEncodedFormEntity entity = new UrlEncodedFormEntity(paramList, CHARSET_UTF_8); entity.setContentType(CONTENT_TYPE_FORM_URL); httpPost.setEntity(entity); } // 执行http请求main response = httpClient.execute(httpPost); resultString = EntityUtils.toString(response.getEntity(), CHARSET_UTF_8); } catch (Exception e) { e.printStackTrace(); } finally { try { if (response != null) { response.close(); } } catch (IOException e) { e.printStackTrace(); } } return resultString; } public static String doPost(String url) { return doPost(url, null); } public static String doPostJson(String url, String json) { // 创建Httpclient对象 CloseableHttpClient httpClient = getHttpClient(); CloseableHttpResponse response = null; String resultString = ""; try { // 创建Http Post请求 HttpPost httpPost = new HttpPost(url); // 创建请求内容 StringEntity entity = new StringEntity(json, ContentType.APPLICATION_JSON); httpPost.setEntity(entity); // 执行http请求 response = httpClient.execute(httpPost); resultString = EntityUtils.toString(response.getEntity(), CHARSET_UTF_8); } catch (Exception e) { e.printStackTrace(); } finally { try { if (response != null) { response.close(); } } catch (IOException e) { e.printStackTrace(); } } return resultString; }} 代码中出现了@Slf4j，作用是引入log，手动打印日志。这个注解是lombok的注解。 解释一下，什么是Route? Route的概念可以理解为客户端机器到目标机器的一条线路，例如使用HttpClient的实现来分别请求 www.163.com 的资源和 www.sina.com 的资源就会产生两个route。缺省条件下对于每个Route，HttpClient仅维护2个连接，总数不超过20个连接。 2.3 笔者着重说一下http连接池 1 为什么要使用http连接池？ 延迟降低，如果不使用连接池，每次发起的http请求都会重新建立tcp连接（三次握手），用完就会关闭连接（4次握手），采用连接池则会减少这不是分时间的消耗。连接池管理的对象都是长连接。 支持更大的并发，由于连接池只适用于请求经常访问同一主机（或同一端口的情况），连接池避免了反复建立连接，抢占端口资源的情况，如果没用连接池，可能导致连接建立不了。 2 设置超时时间 首先要明白三个概念：socketTimeout，connectTimeout，connectionRequestTimeout。 socketTimeout：客户端和服务器读取数据的timeout connectTimeout：客户端和服务器建立连接的timeout connectionRequestTimeout：从连接池获取连接的timeout 3 解释：一次http请求 一次http请求，必定会有三个阶段，一：建立连接；二：数据传送；三，断开连接。 当建立连接在规定的时间内（ConnectionTimeOut ）没有完成，那么此次连接就结束了。后续的SocketTimeOutException就一定不会发生。只有当连接建立起来后， 也就是没有发生ConnectionTimeOutException ，才会开始传输数据，如果数据在规定的时间内(SocketTimeOut)传输完毕,则断开连接。否则，触发SocketTimeOutException。 三、HttpClient的重试机制 上面说了这么多，就是为了引出下面的重试问题。由于项目中要访问外部接口，访问接口的时候，偶尔会出现SocketTimeOutException:Read timed out，其实就是客户端读取服务器的数据超时了。 3.1. 那么问题来了HttpClient有没有重试策略？ 使用PoolingHttpClientConnectionManager得到的InternalHttpClient实例，是抽象类CloseableHttpClient的一个实现。 看一下ClientExecChain接口的实现类

Take a brief look at the build () method

Public CloseableHttpClient build () {/ / omit some code / / add MainClientExec ClientExecChain execChain = this.createMainExec (requestExecCopy, (HttpClientConnectionManager) connManagerCopy, (ConnectionReuseStrategy) reuseStrategyCopy, (ConnectionKeepAliveStrategy) keepAliveStrategyCopy, new ImmutableHttpProcessor (new HttpRequestInterceptor [] {new RequestTargetHost (), new RequestUserAgent (userAgentCopy)}), (AuthenticationStrategy) targetAuthStrategyCopy, (AuthenticationStrategy) proxyAuthStrategyCopy, (UserTokenHandler) userTokenHandlerCopy); execChain = this.decorateMainExec (execChain); / / add ProtocolExec ClientExecChain execChain = new ProtocolExec (execChain, httpprocessorCopy) ClientExecChain execChain = this.decorateProtocolExec (execChain); / / Add request retry executor, if not disabled if (! automaticRetriesDisabled) {HttpRequestRetryHandler retryHandlerCopy = this.retryHandler; if (retryHandlerCopy = = null) {retryHandlerCopy = DefaultHttpRequestRetryHandler.INSTANCE;} execChain = new RetryExec (execChain, retryHandlerCopy) } / / omit part of the code / / if not empty, add ServiceUnavailableRetryExec ServiceUnavailableRetryStrategy serviceUnavailStrategyCopy = this.serviceUnavailStrategy; if (serviceUnavailStrategyCopy! = null) {execChain = new ServiceUnavailableRetryExec ((ClientExecChain) execChain, serviceUnavailStrategyCopy);} / / add RedirectExec if (! this.redirectHandlingDisabled) {authSchemeRegistryCopy = this.redirectStrategy If (authSchemeRegistryCopy = = null) {authSchemeRegistryCopy = DefaultRedirectStrategy.INSTANCE;} execChain = new RedirectExec ((ClientExecChain) execChain, (HttpRoutePlanner) routePlannerCopy, (RedirectStrategy) authSchemeRegistryCopy);} / / omit part of the code return new InternalHttpClient ((ClientExecChain) execChain, (HttpClientConnectionManager) connManagerCopy, (HttpRoutePlanner) routePlannerCopy, cookieSpecRegistryCopy, (Lookup) authSchemeRegistryCopy, (CookieStore) defaultCookieStore, (CredentialsProvider) defaultCredentialsProvider, this.defaultRequestConfig! = null? This.defaultRequestConfig: RequestConfig.DEFAULT, closeablesCopy);}

From top to bottom, different ClientExecChain instances are created. Note: the order in which objects are created is the order of the executor chain.

When constructing a CloseableHttpClient instance, determine whether the automatic retry feature is turned off. AutomaticRetriesDisabled defaults to false. If no executor chain is specified, RetryExec is used. The default retry policy is DefaultHttpRequestRetryHandler.

If the ServiceUnavailableRetryStrategy interface is rewritten, or if DefaultServiceUnavailableRetryStrategy,ServiceUnavailableRetryExec is used, it will also be added to the actuator chain.

By the same token, redirectHandlingDisabled defaults that false,RedirectExec will also be added to the actuator chain and will be executed first.

3.2 execution process

We have seen earlier that the HttiClient we are using is essentially InternalHttpClient, so let's take a look at his method of executing sending data.

@ Override protected CloseableHttpResponse doExecute (final HttpHost target, final HttpRequest request, final HttpContext context) throws IOException, ClientProtocolException {/ / omit some code return this.execChain.execute (route, wrapper, localcontext, execAware);}}

First, call ServiceUnavailableRetryExec's excute () in RedirectExec,RedirectExec, enter ServiceUnavailableRetryExec, call RetryExec's excute (), enter RetryExec, call ProtocolExec's execute (), and finally call MainClientExec's excute ().

At the end of the executor chain, the excute () of HttpRequestExecutor is executed, and the excute () method calls its own doSendRequest ().

Then return step by step and handle the exception when you encounter an exception.

Here is the part of the RetryExec that sends the request

Public CloseableHttpResponse execute (HttpRoute route, HttpRequestWrapper request, HttpClientContext context, HttpExecutionAware execAware) throws IOException HttpException {/ / Parameter check Args.notNull (route, "HTTP route") Args.notNull (request, "HTTP request"); Args.notNull (context, "HTTP context"); / / get all the information of the request header Header [] origheaders = request.getAllHeaders (); / / initialize the number of requests 1 int execCount = 1 While (true) {try {/ / call the underlying executor to execute the http request return this.requestExecutor.execute (route, request, context, execAware) } catch (IOException var9) {/ / when an IO exception occurs, determine whether the context has been interrupted. If interrupted, throw an exception to exit if (execAware! = null & & execAware.isAborted ()) {this.log.debug ("Request has been aborted"); throw var9 } / / judge whether the current execution status should be retried according to the retry strategy If so, enter the following logic if (! this.retryHandler.retryRequest (var9, execCount, context)) {if (var9 instanceof NoHttpResponseException) {NoHttpResponseException updatedex = new NoHttpResponseException (route.getTargetHost (). ToHostString () + "failed to respond") Updatedex.setStackTrace (var9.getStackTrace ()); throw updatedex;} throw var9 } / / if (this.log.isInfoEnabled ()) {this.log.info ("var9.getClass O exception (" + var9.getClass () .log () + ") caught when processing request to" + route + ":" + var9.getMessage ()) } / / if (this.log.isDebugEnabled ()) {this.log.debug (var9.getMessage (), var9) } / / determine whether the current request can repeatedly initiate if (! RequestEntityProxy.isRepeatable (request)) {this.log.debug ("Cannot retry non-repeatable request"); throw new NonRepeatableRequestException ("Cannot retry request with a non-repeatable request entity", var9) } / / set request header request.setHeaders (origheaders); / / log if (this.log.isInfoEnabled ()) {this.log.info ("Retrying request to" + route) } + + execCount;}

When IOException occurs, determine whether to try again. If you retry, record the appropriate number of times. If you do not retry, throw an exception and exit.

/ / for final immutable objects in singleton mode, thread-safe public static final DefaultHttpRequestRetryHandler INSTANCE = new DefaultHttpRequestRetryHandler (); / / number of retries private final int retryCount; / / if a request has been sent successfully, will the exception class private final Set which is not allowed to retry be sent again? private final boolean requestSentRetryEnabled; / /

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.