Shulou(Shulou.com)06/01 Report--

This article will explain in detail the example analysis of nginx+php-fpm service HTTP status code 502. The editor thinks it is very practical, so I share it with you for reference. I hope you can get something after reading this article.

One of our web projects, due to the increase in the number of new cities, resulting in an increase in traffic and DB pressure, as the business side of providing interfaces, has recently received a large number of downstream feedback requests.

502 gateway bad gateway is usually caused by upstream (here is php). For php, the common reason is that the script execution exceeds the timeout setting time, or the timeout setting is too large, so that the php process cannot be released for a long time, and there is no idle worker process to pick up guests.

Our project is caused by the php execution time setting is too short, for this case, we can first appropriately increase the php execution time, first make sure to clear 502, after all, the optimization thing will take more time.

There are two options for controlling the execution time of php: max_execution_time in php.ini and request_terminate_timeout in php-fpm, in which request_terminate_timeout can override max_execution_time, so if you don't want to change the global php.ini, just change the configuration of php-fpm.

Below I will analyze in detail why the execution of the php script exceeds the set time and causes nginx to return 502.

Let's start with the set and repeat the problem:

Nginx and php each start only one worker to facilitate tracking.

The request_terminate_timeout of php-fpm is set to 3s.

Test script test.php

Sleep (20); echo 'ok'

Go go go:

Appears as scheduled after the browser accesses www.v.com/test.php,3S. 404? What???

It's a bad start. Take a look at the configuration file of nginx.

This location configuration jumps to a nice interface when a 5xx error occurs, but I don't have the 50x.html file under / usr/share/nginx/html. So I got a 404. Doesn't that affect the accuracy of my judgment? Just comment it out! Visit again, wait for 3s, and finally the 'normal' interface comes out.

When the environment is good, let's go to the routine below and follow the routine of troubleshooting web problems. First, take a look at the error log:

Nginx:

All errors are reported by recv () failed (104: Connection reset by peer.

Recv failed and the connection was reset. Why is the connection reset? Do not agree with each other.

We are looking at php-fpm 's error log:

(note that the php_admin_ value [error _ log] option in php-fpm specifies the error log of php, which will overwrite the error log in php.ini. But instead of looking at php's mistakes, I'm looking at php-fpm 's mistakes. The error log for php-fpm is specified by the error_log option in php-fpm.conf. )

Each request generates 2 WARNING and 1 NOTICE:

WARNING: script execution timed out and terminated.

WARNING: the child process exits when it receives a SIGTERM signal.

NOTICE: a new child process has been started (because I set pm.min_spare_servers = 1)

It seems that if the worker process of php times out, not only the script execution will be terminated, but also the worker process will exit. It seems that the error connection of nginx was reset because the worker process of php exited (in the case of TCP connection, one party will send RST to the other if it is broken)

You can already know from the log that the execution of the php script timed out and the worker child process exited, causing nginx to report an error Connection reset by peer. Let's take a look at php and nginx through strace:

Php:

1.accept A nginx connection request (socket,bind,listen is completed in master), you can see that the port of nginx is 47039. Reading data from FD0 is from standard input, which is specified by fast-cgi protocol. The connected descriptor after accept is 3.

two。 Read the data passed by nginx from FD3, fastcgi protocol format, received 856 bytes. Why read5 times?

Because the fastcgi protocol packet is 8-byte aligned, it consists of a header and a packet body. And will first send a request packet, including some request ID, version, typpe and other information (header packet body takes 8 bytes each), then send a params packet, pass get parameters and environment variables (packet header 8 bytes, packet body length), and finally send a params packet with no packet body and only packet header, indicating the end of parameter transmission (packet header 8 bytes). So the first three read are used to read the header and body of the request packet, and the header of the params packet, the fourth read is to read the real data, and the last read is to read the header of the last params packet. So the data passed by nginx should be 8 "8" 8 "856" 896 bytes (corresponding to the transmission bytes of the following nginx). Note that if it is in post mode, stdin packets will also be sent.

3. Set hibernation 20s, which is the sleep (20) in the php program, and then the process is terminated, so the rest is gone. The strace program also exited.

Nginx:

For the request from 1.accept to the browser, you can see that the port on the browser side is 56434 and the IP is 192.168.1.105 and the connected FD is 3.

two。 Receive data from FD3, HTTP protocol.

3. Create a socket,FD21 to establish a connection with php.

4. When you connect to FD21, you can see that port 9000 is connected to the machine, where nginx and php-fpm are connected by IP socket, and nginx and php-fpm are deployed on one machine. Unix domain socket can be considered.

5. Write data to FD21 in fast-cgi protocol format. We see that the length of the write is 896, which corresponds to the length received by the php above.

The 6.recvfrom function returns ECONNRESET (Connection reset by peer) from FD21

7. By writing an error message to FD9, you can infer that FD9 is the file descriptor of the nginx error log.

8. Close the connection to the FD21.

9. Writing 502 Bad Gateway to FD3 is the information returned to the browser.

10. By writing an access log to FD8, it can be inferred that FD8 is the file descriptor of the nginx access log.

To verify the inference of the nginx access log and error log. You can see that it is indeed FD8,FD9 and is in write mode.

Well, in this process, we might as well take a look at the transmission of the entire network packet:

Grab the bag through tcpdump, and it is more convenient to see it with an artifact.

Because you only want to see the communication between nginx and php, you know that the port of nginx is 47039, and you can filter out the corresponding packets through tcp.srcport==47039.

You can see the process of data interaction between nginx and php-fpm: 47039-> 9000 establishes a three-way handshake, then sends data to 9000, and 9000 replies to ACK,3S and 9000 replies to RST. No problem.

Note:

SYN,FIN each has a serial number.

ACK,RST does not occupy the serial number (28529 the reqnum and acknum of the two packages are the same)

The serial number is plus 1 per byte (29 packets send 896 bytes, while 29 packets send seq 4219146879, the ack of 30 packets is 4219147775, the difference is exactly 896)

RST does not need a reply.

This is the end of this article on "sample Analysis of nginx+php-fpm Service HTTP status Code 502". I hope the above content can be helpful to you, so that you can learn more knowledge. if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.