Shulou(Shulou.com)06/03 Report--

This article mainly introduces what problems will be encountered when using httpclient, it has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let the editor take you to understand it.

I. Preface

Httpclient is one of the most commonly used tools in java development, and people usually use the more basic api to call remote. Long-term development of crawlers will come into contact with api, which is not commonly used by httpclient, and will encounter all kinds of pits at the same time.

II. Problems and solutions

Question 1:Received fatal alert: handshake_failure

Problem background

When developing a provincial mobile crawler, loading the home page will report a title error, try a variety of methods are not good, and later found that the jdk1.8 will be fine. After a week-long source code search, it was found that the source of the error was that http did not support the encryption algorithm when shaking hands.

256bit (TLS_DHE_RSA_WITH_AES_256_CBC_SHA) is not supported in jdk1.8 versions below

Solution

1. Download the jce expansion package http://www.oracle.com/technetwork/cn/java/javase/downloads/jce-7-download-432124.html

2. Replace the two jar in / jre/lib/security/

3. If you report an error The jurisdiction policy files are not signed by a trusted signer after overwriting, the downloaded version is not correct, and the version corresponding to the jdk version should be downloaded.

Problem 2:Certificates does not conformto algorithm constraints

Problem background

It's wrong to pack the Times with mvn, security.cert.CertificateException: Certificates does not conform toalgorithm constraints

The reason is that in the configuration file after java1.6, it is considered that the security of MD2 encryption is too low, so it does not support this encryption, nor does it support ciphertext with a RSA length less than 1024.

Need to modify JAVA_HOME/jre/lib/security/java.security # jdk.certpath.disabledAlgorithms=MD2, RSA keySize

< 1024 但是这样做需要把每台机器都改一遍，如果新加机器忘记改了，就会继续报错。因此需要一套方法，只在代码层解决问题。 解决方案 经查源码发现了触发问题的代码位置，通过强制继承SSLContextBuilder，并强制把private的keymanagers和trustmanagers的值置空就可以解决这个问题了。 代码如下： static class MySSLContextBuilder extends SSLContextBuilder { static final String TLS = "TLS"; static final String SSL = "SSL"; private String protocol; private Set keymanagers; private Set trustmanagers; private SecureRandom secureRandom; public MySSLContextBuilder() { super(); this.keymanagers = new HashSet(); this.trustmanagers = new HashSet(); }} 问题3：超时时间不生效 问题背景 很多人在使用httpclient时会到网上去找例子，例子中经常会有类似这样的设置 httpGet.getParams().setParameter(ClientPNames.HANDLE_REDIRECTS, !isAutoRelocal); 使用上述方法发送httpclient，在读取配置时，如果发现getParams不为空，则会使得以前设置的所有参数都失效，而使用这里设置的，结果是导致超时时间失效。 解决方案 request.getParams().setParameter是过期方法，其中每一项参数在RequestConfig里都有对应的，遍历出来替换一遍即可。 boolean isRedirect = true; if(request != null) { HttpParams params = request.getParams(); if (params instanceof HttpParamsNames) { // 暂时只支持这个类型 isRedirect = params.getBooleanParameter( ClientPNames.HANDLE_REDIRECTS, true); } // 清空request request.setParams(new BasicHttpParams()); } if(timeOut >

0) {builder = RequestConfig.custom (). SetConnectionRequestTimeout (timeOut) .setConnectTimeout (timeOut) .setSocketTimeout (timeOut) .setRedirectsEnabled (isRedirect) .setCookieSpec (CookieSpecs.BEST_MATCH);} else {builder = RequestConfig.custom (). SetConnectionRequestTimeout (connectionTimeout) .setConnectTimeout (connectionTimeout) .setRedirectsEnabled (isRedirect) .setSocketTimeout (socketTimeout). SetCookieSpec (CookieSpecs.BEST_MATCH);}

Problem 4:fildder snooping problem

Problem background

Developers often use fildder to monitor web requests, but it is difficult to use fildder when using httpclient, and all kinds of methods are not easy to use.

Let's make a mistake for you. Use the following method to solve this problem perfectly and make fildder monitoring easier.

Solution

First of all, the Java side

/ / client builderHttpClientBuilder builder = HttpClients.custom (); if (useFidder) {/ / default fidder write dead builder.setProxy (new HttpHost ("127.0.0.1", 8888);}

Fildder end

Tools- > fiddler options- > https- > actions- > export root certificate to.\ bin\ keytool.exe-import-file C:\ Users\\ Desktop\ FiddlerRoot.cer-keystore FiddlerKeystore-alias Fiddler

Question 5: support for gzip

Problems and solutions

Some websites return gzip compression, and the returned content is the result of compression and needs to be decompressed.

The code is as follows:

HttpClient wrappedHttpClient = builder.setUserAgent (requestUA) .addInterceptorLast (new HttpResponseInterceptor () {@ Override public void process (HttpResponse httpResponse, HttpContext httpContext) throws HttpException, IOException {HttpEntity httpEntity = httpResponse.getEntity (); Header header = httpEntity.getContentEncoding () If (header! = null) {for (HeaderElement element: header.getElements ()) {if ("gzip" .equals IgnoreCase (element.getName () {httpResponse.setEntity (new GzipDecompressingEntity (httpResponse.getEntity () }}) Thank you for reading this article carefully. I hope the article "what problems you will encounter when using httpclient" shared by the editor will be helpful to you. At the same time, I also hope you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.