Shulou(Shulou.com)06/01 Report--

How to use Nginx as a HTTPS forward proxy server, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain in detail for you, people with this need can come to learn, I hope you can gain something.

NGINX is mainly designed as a reverse proxy server, but with the development of NGINX, it can also be used as one of the options for forward proxy. Forward agent itself is not complex, and how to proxy encrypted HTTPS traffic is the main problem that forward agent needs to solve. This article will introduce two schemes of using NGINX to forward proxy HTTPS traffic, as well as their usage scenarios and main problems.

Classification of forward agents in HTTP/HTTPS

A brief introduction to the classification of forward agents as background for understanding the following:

Classified by whether the client is aware or not

General proxy: on the client side, you need to manually set the address and port of the agent in the browser or in the system environment variable. For example, squid, specify the squid server IP and port 3128 on the client.

Transparent proxy: the client does not need to make any proxy settings, and the role of "proxy" is transparent to the client. Such as Web Gateway devices in corporate network links.

Classification by whether the agent decrypts the HTTPS

Tunnel agent: that is, transparent transmission agent. The proxy server only transmits HTTPS traffic over the TCP protocol and does not perceive the specific content of its proxy traffic without decryption. The client does direct TLS/SSL interaction with the destination server it accesses. The NGINX proxy approach discussed in this article belongs to this pattern.

Man in the middle (MITM, Man-in-the-Middle) proxy: the proxy server decrypts the HTTPS traffic, uses the self-signed certificate to complete the TLS/SSL handshake to the client, and completes the normal TLS interaction to the destination server. Establish two TLS/SSL sessions in the client-agent-server link.

Note: in this case, the client actually gets the self-signed certificate of the proxy server during the TLS handshake phase. The verification of the certificate chain is not successful by default, so you need to trust the Root CA certificate of the self-signed certificate of the proxy at the client. So it is felt by the client in the process. If you want to become a transparent proxy without feeling, you need to push the self-built Root CA certificate to the client, which can be realized in the internal environment of the enterprise.

Why do forward agents need special treatment to handle HTTPS traffic?

When acting as a reverse proxy, the proxy server usually terminates (terminate) HTTPS encrypted traffic and forwards it to the backend instance. The process of encryption, decryption and authentication of HTTPS traffic occurs between the client and the reverse proxy server.

As a forward proxy, when handling the traffic sent by the client, the HTTP is sealed in the TLS/SSL, and the proxy server cannot see the domain name that the client requests URL to access, as shown in the following figure. So proxy HTTPS traffic, compared with HTTP, needs to do some special processing.

The solution of NGINX

According to the previous classification, the way NGINX solves the HTTPS proxy belongs to the transparent (tunnel) mode, that is, it does not decrypt and does not perceive the upper layer traffic. The specific ways are as follows: 7-tier and 4-tier solutions.

HTTP CONNECT Tunnel (layer 7 solution)

Historical background

As early as 1998, when there was no formal birth of TLS in the SSL era, Netscape, which dominates the SSL protocol, put forward the INTERNET-DRAFT about using web agents to tunneling SSL traffic. The core idea is to establish a HTTP CONNECT Tunnel between the client and the agent by using HTTP CONNECT request. In the CONNECT request, the destination host and port that the client needs to access need to be specified. The original image in Draft is as follows:

For the entire process, you can refer to the figure in the HTTP authoritative guide:

The client sends a HTTP CONNECT request to the proxy server.

The proxy server uses the host and port in the HTTP CONNECT request to establish an TCP connection with the destination server.

The proxy server returns a HTTP 200response to the client.

The client and the proxy server establish a HTTP CONNECT tunnel. After the HTTPS traffic reaches the proxy server, it is directly transmitted to the remote destination server through TCP. The role of the proxy server is to pass through HTTPS traffic and does not need to decrypt HTTPS.

NGINX ngx_http_proxy_connect_module module

As a reverse proxy server, NGINX has not officially supported the HTTP CONNECT method. However, based on the modularization and extensibility of NGINX, Ali's @ chobits provides a ngx_http_proxy_connect_module module to support the HTTP CONNECT method, so that NGINX can be extended to a forward proxy.

Environment building

Take the environment of CentOS 7 as an example.

1) installation

For the newly installed environment, refer to the normal installation steps and the steps for installing this module (https://github.com/chobits/ngx_http_proxy_connect_module)). After typing the corresponding version of patch, add the parameter-add-module=/path/to/ngx_http_proxy_connect_module when configure. Examples are as follows:

/ configure-- user=www-- group=www-- prefix=/usr/local/nginx-- with-http_ssl_module-- with-http_stub_status_module-- with-http_realip_module-- with-threads-- add-module=/root/src/ngx_http_proxy_connect_module

For the environment that has been installed, compiled and installed, you need to add the above modules as follows:

# stop NGINX service # systemctl stop nginx # back up the original execution file # cp / usr/local/nginx/sbin/nginx / usr/local/nginx/sbin/nginx.bak # recompile # cd / usr/local/src/nginx-1.16.0. / configure-- user=www-- group=www-- prefix=/usr/local/nginx-- with-http_ssl_module-- with-http_stub_status_module- -with-http_realip_module-- with-threads-- add-module=/root/src/ngx_http_proxy_connect_module # make # do not make install # overwrite the newly generated executable copy to the original nginx executable file # cp objs/nginx / usr/local/nginx/sbin/nginx # / usr/bin/nginx-V nginx version: nginx/1.16.0 built by gcc 4.8.5 20150623 (Red Hat 4.8.5- 36) (GCC) built with OpenSSL 1.0.2k-fips 26 Jan 2017 TLS SNI support enabled configure arguments:-user=www-- group=www-- prefix=/usr/local/nginx-- with-http_ssl_module-- with-http_stub_status_module-- with-http_realip_module-- with-threads-- add-module=/root/src/ngx_http_proxy_connect_module

2) nginx.conf file configuration

Server {listen 443; # dns resolver used by forward proxying resolver 114.114.114.114; # forward proxy for CONNECT request proxy_connect; proxy_connect_allow 443; proxy_connect_connect_timeout 10s; proxy_connect_read_timeout 10s; proxy_connect_send_timeout 10s; # forward proxy for non-CONNECT request location / {proxy_pass http://$host; proxy_set_header Host $host;}}

Working with scen

Layer 7 needs to establish a tunnel through HTTP CONNECT, which belongs to the client-aware common proxy mode, and the HTTP (S) proxy server IP and port need to be manually configured on the client. On the client side, use curl plus-x parameter to access as follows:

# curl https://www.baidu.com-svo / dev/null-x 39.105.196.164 About to connect () to proxy 39.105.196.164 port 443 (# 0) * Trying 39.105.196.164 * Connected to 39.105.196.164 (39.105.196.164) port 443 (# 0) * Establish HTTP proxy tunnel to www.baidu.com:443 > CONNECT www.baidu.com:443 HTTP/1.1 > Host: www.baidu.com:443 > User-Agent: curl/7.29.0 > Proxy-Connection: Keep-Alive >

< HTTP/1.1 200 Connection Established < Proxy-agent: nginx < * Proxy replied OK to CONNECT request * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 * Server certificate: * subject: CN=baidu.com,O="Beijing Baidu Netcom Science Technology Co., Ltd",OU=service operation department,L=beijing,ST=beijing,C=CN ... >

GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: www.baidu.com > Accept: * / * >

< HTTP/1.1 200 OK ... { [data not shown] 从上面-v参数打印出的细节，可以看到客户端先往代理服务器39.105.196.164建立了HTTP CONNECT隧道，代理回复HTTP/1.1 200 Connection Established后就开始交互TLS/SSL握手和流量了。 NGINX stream (4层解决方案) 既然是使用透传上层流量的方法，那可不可做成"4层代理"，对TCP/UDP以上的协议实现彻底的透传呢?答案是可以的。NGINX官方从1.9.0版本开始支持ngx_stream_core_module模块，模块默认不build，需要configure时加上--with-stream选项来开启。 问题 用NGINX stream在TCP层面上代理HTTPS流量肯定会遇到本文一开始提到的那个问题：代理服务器无法获取客户端想要访问的目的域名。因为在TCP的层面获取的信息仅限于IP和端口层面，没有任何机会拿到域名信息。要拿到目的域名，必须要有拆上层报文获取域名信息的能力，所以NGINX stream的方式不是完全严格意义上的4层代理，还是要略微借助些上层能力。 ngx_stream_ssl_preread_module模块 要在不解密的情况下拿到HTTPS流量访问的域名，只有利用TLS/SSL握手的***个Client Hello报文中的扩展地址SNI (Server Name Indication)来获取。NGINX官方从1.11.5版本开始支持利用ngx_stream_ssl_preread_module模块来获得这个能力，模块主要用于获取Client Hello报文中的SNI和ALPN信息。对于4层正向代理来说，从Client Hello报文中提取SNI的能力是至关重要的，否则NGINX stream的解决方案无法成立。同时这也带来了一个限制，要求所有客户端都需要在TLS/SSL握手中带上SNI字段，否则NGINX stream代理完全没办法知道客户端需要访问的目的域名。 环境搭建 1) 安装 对于新安装的环境，参考正常的安装步骤，直接在configure的时候加上--with-stream，--with-stream_ssl_preread_module和--with-stream_ssl_module选项即可。示例如下： ./configure --user=www --group=www --prefix=/usr/local/nginx --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-threads --with-stream --with-stream_ssl_preread_module --with-stream_ssl_module 对于已经安装编译安装完的环境，需要加入以上3个与stream相关的模块，步骤如下： # 停止NGINX服务 # systemctl stop nginx # 备份原执行文件 # cp /usr/local/nginx/sbin/nginx /usr/local/nginx/sbin/nginx.bak # 在源代码路径重新编译 # cd /usr/local/src/nginx-1.16.0 # ./configure --user=www --group=www --prefix=/usr/local/nginx --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-threads --with-stream --with-stream_ssl_preread_module --with-stream_ssl_module # make # 不要make install # 将新生成的可执行文件拷贝覆盖原来的nginx执行文件 # cp objs/nginx /usr/local/nginx/sbin/nginx # nginx -V nginx version: nginx/1.16.0 built by gcc 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) built with OpenSSL 1.0.2k-fips 26 Jan 2017 TLS SNI support enabled configure arguments: --user=www --group=www --prefix=/usr/local/nginx --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-threads --with-stream --with-stream_ssl_preread_module --with-stream_ssl_module 2) nginx.conf文件配置 NGINX stream与HTTP不同，需要在stream块中进行配置，但是指令参数与HTTP块都是类似的，主要配置部分如下： stream { resolver 114.114.114.114; server { listen 443; ssl_preread on; proxy_connect_timeout 5s; proxy_pass $ssl_preread_server_name:$server_port; } } 使用场景 对于4层正向代理，NGINX对上层流量基本上是透传，也不需要HTTP CONNECT来建立隧道。适合于透明代理的模式，比如将访问的域名利用DNS解定向到代理服务器。我们可以通过在客户端绑定/etc/hosts来模拟。 在客户端： cat /etc/hosts ... # 把域名www.baidu.com绑定到正向代理服务器39.105.196.164 39.105.196.164 www.baidu.com # 正常利用curl来访问www.baidu.com即可。 # curl https://www.baidu.com -svo /dev/null * About to connect() to www.baidu.com port 443 (#0) * Trying 39.105.196.164... * Connected to www.baidu.com (39.105.196.164) port 443 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 * Server certificate: * subject: CN=baidu.com,O="Beijing Baidu Netcom Science Technology Co., Ltd",OU=service operation department,L=beijing,ST=beijing,C=CN * start date: 5月 09 01:22:02 2019 GMT * expire date: 6月 25 05:31:02 2020 GMT * common name: baidu.com * issuer: CN=GlobalSign Organization Validation CA - SHA256 - G2,O=GlobalSign nv-sa,C=BE >

GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: www.baidu.com > Accept: * / * >

< HTTP/1.1 200 OK < Accept-Ranges: bytes < Cache-Control: private, no-cache, no-store, proxy-revalidate, no-transform < Connection: Keep-Alive < Content-Length: 2443 < Content-Type: text/html < Date: Fri, 21 Jun 2019 05:46:07 GMT < Etag: "5886041d-98b" < Last-Modified: Mon, 23 Jan 2017 13:24:45 GMT < Pragma: no-cache < Server: bfe/1.0.8.18 < Set-Cookie: BDORZ=27315; max-age=86400; domain=.baidu.com; path=/ < { [data not shown] * Connection #0 to host www.baidu.com left intact 常见问题 1) 客户端手动设置代理导致访问不成功 4层正向代理是透传上层HTTPS流量，不需要HTTP CONNECT来建立隧道，也就是说不需要客户端设置HTTP(S)代理。如果我们在客户端手动设置HTTP(s)代理是否能访问成功呢? 我们可以用curl -x来设置代理为这个正向服务器访问测试，看看结果： # curl https://www.baidu.com -svo /dev/null -x 39.105.196.164:443 * About to connect() to proxy 39.105.196.164 port 443 (#0) * Trying 39.105.196.164... * Connected to 39.105.196.164 (39.105.196.164) port 443 (#0) * Establish HTTP proxy tunnel to www.baidu.com:443 >

CONNECT www.baidu.com:443 HTTP/1.1 > Host: www.baidu.com:443 > User-Agent: curl/7.29.0 > Proxy-Connection: Keep-Alive > * Proxy CONNECT aborted * Connection # 0 to host 39.105.196.164 left intact

You can see that the client attempts to establish the HTTP CONNECT tunnel before the forward NGINX, but because the NGINX is transparently transmitted, the CONNECT request is forwarded directly to the destination server. The destination server does not accept the CONNECT method, so eventually "Proxy CONNECT aborted" occurs, resulting in unsuccessful access.

2) the access is not successful because the client does not have SNI.

One of the key factors mentioned above in using NGINX stream as a forward proxy is to extract the SNI field from the Client Hello using ngx_stream_ssl_preread_module. If the client does not carry the SNI field, it will cause the proxy server to be unable to know the destination domain name, resulting in unsuccessful access.

In transparent proxy mode (simulated by manually binding hosts), we can simulate with openssl on the client side:

# openssl s_client-connect www.baidu.com:443-msg CONNECTED (00000003) > TLS 1.2 [length 0005] 16 03 01 011c > TLS 1.2 Handshake [length 011c] ClientHello 01 00 01 18 03 03 6b 2e 75 86 52 6c d5 a5 80 d7 a4 61 65 6d 72 53 33 fb 33 f0 43 a3 aa c2 4a e3 47 84 9f 69 8b d6 00 00 ac c0 30 c0 2c c0 28 c0 24 c0 14 c0 0a 00 a5 00 a3 00 a1 00 9f 00 6b 00 6a 00 69 00 68 00 39 00 38 00 37 00 36 00 88 00 87 00 86 00 85 c0 32 c0 2e c0 2a c0 26 c0 0f c0 05 00 9d 00 3d 00 35 00 84 c0 2f c0 2b c0 27 c0 23 c0 13 c0 09 00 a4 00 a2 00 a0 00 9e 00 67 00 40 00 3f 00 3e 00 33 00 32 00 31 00 30 00 9a 00 99 00 98 00 97 00 45 00 44 00 43 00 42 c0 31 c0 2d c0 29 c0 25 c0 0e c0 04 00 9c 00 3c 00 2f 00 96 00 41 c0 12 c0 08 00 16 00 13 00 10 00 0d c0 0d c0 03 00 0a 00 07 c0 11 c0 07 c0 0c c0 02 00 05 00 04 00 ff 01 00 00 43 00 0b 00 04 03 00 01 02 00 0a 00 0a 00 08 00 17 00 19 00 18 00 16 00 23 00 00 00D 00 20 00 1e 06 01 06 02 06 03 05 05 02 03 04 01 04 02 04 03 01 03 03 03 02 03 02 02 02 03 00 f 00 01 140285606590352:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:-no peer certificate available-No client certificate CA names sent-SSL handshake has read 0 bytes and written 289 bytes.

Openssl s_client does not have SNI by default. You can see that the above request ends during the TLS/SSL handshake phase and ends after the Client Hello is issued. Because the proxy server does not know which destination domain name to forward the Client Hello to.

If you use openssl with the servername parameter to specify SNI, you can access it successfully, as follows:

# openssl s_client-connect www.baidu.com:443-servername www.baidu.com is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.