Shulou(Shulou.com)05/31 Report--

The content of this article mainly focuses on why there is such a big delay after the HTTP call. The content of the article is clear and clear. It is very suitable for beginners to learn and is worth reading. Interested friends can follow the editor to read together. I hope you can get something through this article!

1. Background

Recently, the project test encountered a strange phenomenon. In the test environment, the average time spent invoking the back-end HTTP service through Apache HttpClient is close to 39.2ms. Maybe you don't think it's normal at first glance. What's so strange about it? In fact, let me talk about some basic information. The back-end HTTP service does not have any business logic, but converts a string to uppercase and returns. The string length is only 100 characters, and the network ping latency is only about 1.9ms. So, theoretically, the call should take about 2-3ms, but why does it take an average of 39.2ms?

Due to work reasons, the problem of time-consuming invocation is not uncommon to me. I often help businesses solve the problem of internal RPC framework call timeout, but HTTP calls take * times. However, the routine for troubleshooting problems is the same. The main methodology is nothing more than the investigation methods from the outside to the inside and from top to bottom. Let's first take a look at some indicators in the periphery to see if we can find clues.

two。 Peripheral index

2.1 system indicators

Mainly look at some peripheral system indicators (note: both the called and the called machine should be looked at). For example, load, CPU. It only takes a top command to get a panoramic view.

Therefore, make sure that the CPU and the load are free. Since there was no screenshot at that time, the picture will not be shown here.

2.2 process indicators

The process index of Java program mainly depends on GC and thread stack (note: both the called machine and the called machine should be looked at).

Young GC is very small and time-consuming is within 10ms, so there is no long-term STW.

Because the average call time 39.2ms is relatively large, if the time-consuming is caused by the code, the thread stack should be able to find something. After looking at it, I found nothing, and the main manifestation of the thread stack of the service is that the thread of the thread pool is waiting for the task, which means that the thread is not busy.

Do you feel like you're running out of skills? what should we do next?

3. Local recurrence

If the local (local MAC system) can be reproduced, it is also excellent for troubleshooting problems.

So I wrote a simple Test program using Apache HttpClient locally, called the back-end HTTP service directly, and found that the average time spent was about 55ms. Why, it's a little different from the result of the test environment 39.2ms. Mainly the HTTP service machines of the local and the back-end of the test environment are cross-region, and the ping latency is about 26ms, so the delay has increased. However, there are local problems, because the ping latency is 26ms, the back-end HTTP service logic is simple, almost no time-consuming, so the average local call time should be about 26ms, why 55ms?

Are you getting more and more confused and confused and don't know how to do it?

During the period, I wondered if there was something wrong with the use of Apache HttpClient, so I wrote a simple program using HttpURLConnection that comes with JDK. I tested it and the result was the same.

4. Diagnosis

4.1 location

In fact, from the external system indicators, process indicators, as well as local recurrence, it can be concluded that it is not a procedural reason. What about the TCP protocol level?

Students with experience in network programming must know what parameters of TCP will cause this phenomenon. Yes, you guessed right. It's TCP_NODELAY.

Which side of the program is not set up, the caller or the callee?

The caller uses Apache HttpClient, and the default setting for tcpNoDelay is true. Let's take a look at the callee, our back-end HTTP service, which uses HttpServer that comes with JDK.

HttpServer server = HttpServer.create (new InetSocketAddress (config.getPort ()), BACKLOGS)

Unexpectedly did not see the direct setting of the tcpNoDelay interface, turned over the source code. Oh, it turns out that there is this static block in the class of ServerConfig, which is used to get the startup parameters. The default ServerConfig.noDelay is false.

Static {AccessController.doPrivileged (new PrivilegedAction () {public Void run () {ServerConfig.idleInterval = Long.getLong ("sun.net.httpserver.idleInterval", 30L) * 1000L; ServerConfig.clockTick = Integer.getInteger ("sun.net.httpserver.clockTick", 10000); ServerConfig.maxIdleConnections = Integer.getInteger ("sun.net.httpserver.maxIdleConnections", 10000); ServerConfig.drainAmount = Long.getLong ("sun.net.httpserver.drainAmount", 65536L) ServerConfig.maxReqHeaders = Integer.getInteger ("sun.net.httpserver.maxReqHeaders", 200); ServerConfig.maxReqTime = Long.getLong ("sun.net.httpserver.maxReqTime",-1L); ServerConfig.maxRspTime = Long.getLong ("sun.net.httpserver.maxRspTime",-1L); ServerConfig.timerMillis = Long.getLong ("sun.net.httpserver.timerMillis", 1000L); ServerConfig.debug = Boolean.getBoolean ("sun.net.httpserver.debug"); ServerConfig.noDelay = Boolean.getBoolean ("sun.net.httpserver.nodelay") Return null;});}

4.2 Verification

In the back-end HTTP service, add the startup "- Dsun.net.httpserver.nodelay=true" parameter, and try again. The effect is obvious, and the average time spent is reduced from 39.2ms to 2.8ms.

The problem is solved, but if you stop here, it will be too cheap in this case. Because there are still a lot of doubts waiting for you?

Why is the latency reduced from 39.2ms to 2.8ms by adding TCP_NODELAY?

Why is the average latency of local tests 55ms rather than the latency 26ms of ping?

How exactly does the TCP protocol send packets?

Come on, let's take advantage of the hot iron.

5. Solve the puzzle

Who is TCP_NODELAY?

In Socket programming, the TCP_NODELAY option is used to control whether the Nagle algorithm is turned on. In Java, turn off the Nagle algorithm for ture and turn on Nagle algorithm for false. You must ask what is the Nagle algorithm?

5.2 what is the Nagle algorithm?

Nagle algorithm is a method to improve the efficiency of TCP/IP network by reducing the number of packets sent over the network. It is named after the inventor John Nagle, which John Nagle used in 1984 to try to solve the Ford Motor Company's network congestion problem.

Imagine if the application generates 1 byte of data at a time, and then the 1 byte of data is sent to the remote server in the form of network packets, then it is easy to cause the network to overload due to too many packets. In this typical case, transmitting a packet with only 1 byte of valid data costs an additional overhead of 40 bytes of long header (that is, 20 bytes of IP header + 20 bytes of TCP header), and the utilization of this payload (payload) is extremely low.

The content of the Nagle algorithm is relatively simple, the following is pseudo code:

If there is new data to send if the window size > = MSS and available data is > = MSS send complete MSS segment now else if there is unconfirmed data still in the pipe enqueue data in the buffer until an acknowledge is received else send data immediately end if end if end if

The specific measures are as follows:

If the content sent is greater than or equal to 1 MSS, send it immediately

If there is no previous packet that has not been ACK, send it immediately

If there is a previous packet that has not been ACK, cache the content sent

If an ACK is received, the cached content is sent immediately. (MSS is the * data segment that TCP packets can transmit each time)

5.3 what is Delayed ACK?

As we all know, in order to ensure the reliability of transmission, the TCP protocol stipulates that an acknowledgment needs to be sent to the other party when a packet is received. It is expensive to simply send an acknowledgment (20 bytes of IP header + 20 bytes of TCP header). TCP Delayed ACK (delayed acknowledgement) is an effort to improve network performance to solve this problem. It combines several ACK responses into a single response, or sends ACK responses and response data to each other, thus reducing protocol overhead.

The specific measures are as follows:

When there is response data to be sent, ACK will send it to the other party immediately with the response data.

If there is no response data, ACK will delay sending, waiting to see if there is any response data that can be sent together. On Linux systems, the default delay time is 40ms

If the other party's second packet arrives while waiting for the ACK to be sent, the ACK should be sent immediately. However, if three packets of the other party arrive one after another, whether ACK will be sent immediately when the third segment arrives depends on the above two.

5.4 what chemical reaction will take place between Nagle and Delayed ACK?

Both Nagle and Delayed ACK can improve the efficiency of network transmission, but together they will do bad things with good intentions. For example, the following scenario:

An and B carry on data transmission: a runs Nagle algorithm, B runs Delayed ACK algorithm.

If A sends a packet to B, B will not respond immediately because of Delayed ACK. While A uses the Nagle algorithm, A will wait until B's ACK,ACK does not come and will not send the second packet. If the two packets are responding to the same request, then the request will be delayed 40ms.

5.5 grab a bag and play with it

Let's grab a package for verification. Execute the following script on the back-end HTTP service to easily complete the capture process.

Sudo tcpdump-I eth0 tcp and host 10.48.159.165-s 0-w traffic.pcap

As shown in the figure below, this is a display of using Wireshark to analyze the contents of the package. In the red box, there is a complete POST request processing process. See the difference between 40ms (0.1859-0.1448 = 0.0411s = 41ms) between the 130th and 149th serial numbers. This is the chemical reaction sent by Nagle and Delayed ACK, in which 10.48.159.165 runs Delayed ACK,10.22.29.180 and Nagle algorithm. 10.22.29.180 is waiting for ACK, while 10.48.159.165 triggers Delayed ACK, thus foolishly waiting for 40ms.

This explains why the test environment is 39.2ms, because most of it is delayed by Delayed ACK's 40ms.

But when it comes to local reproduction, why is the average delay of the local test 55ms, rather than the latency 26ms of ping? Let's grab a bag, too.

As shown in the figure below, there is a complete POST request processing process in the red box. The difference between sequence number 8 and sequence number 9 is about 25ms, and then minus the network delay is about half the 13ms of ping delay, so Delayed Ack is about 12ms (because there is some difference between MAC system and Linux locally).

1. Linux uses / proc/sys/net/ipv4/tcp_delack_min as the system configuration to control the time of Delayed ACK. Linux defaults to 40ms.

2. MAC controls Delayed ACK through net.inet.tcp.delayed_ack system configuration.

Delayed_ack=0 responds after every packet (OFF) delayed_ack=1 always employs delayed ack, 6 packets can get 1 ack delayed_ack=2 immediate ack after 2nd packet, 2 packets per ack (Compatibility Mode) delayed_ack=3 should auto detect when to employ delayed ack, 4packets per ack. (DEFAULT) set to 0 to disable delayed ACK, 1 to always delay ACK, 2 to reply to one ACK every two packets, and 3 to automatically detect the timing of replying to ACK.

5.6 Why can TCP_NODELAY solve the problem?

TCPNODELAY turns off the Nagle algorithm so that even if the ACK of the previous packet does not arrive, the next packet is sent, thus breaking the impact of Delayed ACK. Generally, in network programming, it is strongly recommended to turn on TCPNODELAY to improve the response speed.

Of course, the problem can also be solved through the configuration of Delayed ACK-related systems, but it is not recommended because it is inconvenient to modify the machine configuration.

Thank you for your reading. I believe you have a certain understanding of the question of "why the delay is so large after the HTTP call". Go ahead and practice it. If you want to know more about it, you can follow the website! The editor will continue to bring you better articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.