Shulou(Shulou.com)06/02 Report--

This article mainly explains "the troubleshooting process of a large number of nginx 502 abnormal errors". The explanation in this article is simple and clear, easy to learn and understand. Please follow the ideas of Xiaobian and go deep into it slowly to study and learn "the troubleshooting process of a large number of nginx 502 abnormal errors" together.

Problem background: 118 machine interface failure alarm, a period of time after recovery

Process of investigation:

1. Confirmation problem: nginx has a large number of 502 exceptions, that is, the backend service corresponding to nginx may process slowly or discard the request.

2. Check the CPU, memory, network and other basic indicators of the 118 machine. It is found that the inlet traffic has dropped sharply during this period of eth0, almost 0, and the problem has occurred for a few days. The time coincides with other indicators. It indicates that the traffic reduction must be a major breakthrough in the problem.

3. eth0 network ingress traffic is 0, there are two possibilities: 1. network problems cause the machine to fail;2. tomcat service load is too high, causing new socket request connection to be rejected;

4, check the network: found that the ping detection of this machine without packet loss phenomenon, in addition, the other machines in the same room are also no packet loss situation, then the network problem can be ruled out.

5, tomcat problem: Because the acceptcount has been adjusted to 10000 before (when the number of connections is about 20000, it will lead to a long waiting time for requests in the queue), the first reaction is: this time period, there must be some requests that cause tomcat's request queue to be full, resulting in a queue of requests, which takes a long time. Other requests are dropped because the queue is full, and tomcat does not accept TCP connections, resulting in reduced ingress traffic. So, add request time %T to tomcat log, hoping to find requests with long request times in the time period in question.

6: Waiting for a day.

7. Looking at yesterday's data: I found that in the same time period, there were many simple requests that took nearly 2 minutes, and many took nearly 1 minute.

Analysis of the command: sort-k11rgfile.txt: Simple explanation: according to the space division, division according to the 11th column of data sorting, -r is the reverse sort, -g is to convert the string to numerical sorting

8. Find the corresponding request address, as well as the client, find the corresponding person in charge of the client, and communicate with him. Discovery: This interface will be called in large quantities every morning.

Thank you for reading, the above is the content of "nginx a large number of 502 abnormal error troubleshooting process", after the study of this article, I believe that everyone has a deeper understanding of the problem of nginx a large number of 502 abnormal error troubleshooting process, the specific use of the situation also needs to be verified by practice. Here is, Xiaobian will push more articles related to knowledge points for everyone, welcome to pay attention!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.