Shulou(Shulou.com)06/01 Report--

This article introduces how to achieve file download breakpoint continuation and what are the basic features of TCP, the content is very detailed, interested friends can refer to, hope to be helpful to you.

Start with an image transmission project

The success rate of establishing a network is an important index of the network, especially the wireless network. For client-side demo, no matter which platform makes network request and calls socket interface, it can complete the basic data sending and receiving process. However, without optimization, the success rate is basically around 95%. The system call interface is basically suitable for the network speed of 10Mbps in the era of wired network. In order to improve the success rate, there must be a good underlying design.

The interviewee has delivered the business related to Taobao ARbuy. I believe he has a general knowledge of the technical process with relevant project experience. The core of the technology is object recognition. If the recognition model is placed on the APP side, the rest is to control the interaction of messages, improve the success rate of packet signaling, basically every big factory, Wechat, Taobao, Alipay, Meituan and so on all have their own routines. If the picture is put to the server for identification, then the upload of the data packet needs to be specially designed, not just a call to the network interface.

Several basic ideas

Compared with the wired network, the most important feature of the wireless network is the high delay and low bandwidth, and the network state can not be predicted. Let's first take a look at the network request process: DNS query, TCP connection, TLS handshake, data transmission, ack response. About DNS optimization and TLS handshake 0rtt is free to expand, there are too many contents, here we will talk about data transmission and response timeout design. Generally speaking, the most intuitive experience is the breakpoint continuation design of Xunlei downloading large files, and the respondent has also implemented the FTP/TFTP protocol before, focusing on the design and implementation of the continuation biography package. The data continuation of each major client can be divided into the following categories:

1. Query before re-transmission

After failing to send a large data stream, query the location of the receiving point on the server first, get the return packet, and then resume the transmission from the next serial number.

There needs to be a signaling request, usually using a RPC request.

2. Reply and confirm when uploading, and directly resume the upload when reconnected after disconnection. Representative: the famous Wechat.

The implementation of this protocol code is complicated, including the time interval of ACK response packets, the number of data streams received to respond, and so on, which need to be adjusted according to the network condition. You can refer to the nagle algorithm and piggyback ACK features in the TCP protocol for basic reference.

3. After the disconnection and re-establishment, the client directly retransmits, the server informs the client what is missing, and the client retransmits it quickly. Representatives: Taobao, Tmall

The design of the protocol is perfect, but if you knock on the code and debug it, you will know which holes there are. You can think about it first.

Service selection

Breakpoint continuation of the basic methods are the above, choose a good design timeout, you can make a better plan.

Breakpoint continuation is troublesome because after the client sends the packet, it is impossible to know how much the server has received. The above solutions solve this problem from different ideas.

Let's analyze three design schemes. The first one has an extra rtt, which is not a suitable choice in a weak network environment. At first glance, the third scheme is better designed, but the implementation of TCP in the kernel has a buffer, which can be viewed through getsockopt, which defaults to 16k. For files uploaded of about 16k, the client can throw them into the buffer. If the connection is disconnected, solution 3 will be downgraded to send an empty packet first, waiting for the server to return the packet to tell the client where it has been received. Yes, it is plan one, which adds an rtt out of thin air. And if the timeout design is not good, the overall effect is not as good as the first plan.

Look at the second plan, the biggest feature of the wireless network is large jitter, through the server timely and continuous response to the client to confirm, design appropriate timeout, you can timely identify whether the TCP has become a false connection (NAT table failure, etc.). Of course, for large files such as Xunlei download, plan 3 is appropriate. For social scenarios such as Wechat and other IM, scheme 2 is suitable for those that require high immediacy.

Basic characteristics of TCP

Let's take a look at TCP's three-way handshake:

So why doesn't server need to confirm message 3? In other words, how does client know that the ack message has been received by the peer? Here is a convention: TCP only acknowledges valid data (including SYN messages and FIN messages), not ack messages. That is, if message 3 is lost, it will not affect the subsequent communication between the two parties, and this is not valid information, so there will be a piggyback ACK feature in TCP, that is, ACK will be sent to the peer together with valid application layer data. If there is no data exchange within a period of time, 200ms will return a separate ACK message when it times out. So message 2 is missing? The server will start the timer after sending the SYNACK message, and the message 3 will be retransmitted if it does not receive the message after the timeout. Here is a SYN message flood attack, each has its own precautions, Taobao not long ago because of the underground industry Shangxixi police station reported the case. Some servers use syncookie to prevent illegal clients, but it has a certain impact on performance, which has to be combined with business optimization. Server backend defense against ping, ICMP and DDOS attacks will be carried out when available.

Here you can consider such a scenario: an agrees with B to go to the classroom for self-study at two o'clock in the afternoon, A sends the message to B (message 1), B responds to the message that it has received the message (message 2), and A responds to B to confirm that the response has been received (message 3). Let's see, how can A confirm that message 3 has been sent to B? A confirmation is not required, because if B does not receive message 3, it will continue to retransmit message 2. How does B tell A that he has received message 3? Similarly, there is no need to confirm, as long as the data is sent directly. In this example, An and B go directly to the classroom for self-study. Message 2 confirms that the information status of the two sides has been synchronized at this time.

Slow start and congestion avoidance

The sender maintains a state variable called congestion window cwnd (congestion window). The size of the congestion window depends on the degree of congestion of the network and is changing dynamically. The sender makes his sending window equal to the congestion window, and considering the receiving ability of the receiver, the sending window may be smaller than the congestion window.

Slow start is worth is that when a TCP link is first established, do not send a large amount of data to cause a surge of network congestion, but gradually increase the congestion window according to the feedback. The following is the packet sending behavior of the client when the connection is first established, regardless of the delay of ACK at the receiver:

You can see that the congestion window is growing by multiplier.

In order to prevent network congestion caused by excessive growth of cwnd, it is also necessary to set a threshold ssthresh variable. When cwnd > ssthresh, congestion avoidance algorithm should be adopted. The congestion avoidance algorithm is that every time the receiver receives an ack, it adds 1 to the cwnd.

Fast retransmission

When the sender receives three repeated acknowledgements, it can judge that the peer has not received the subsequent message of the acknowledgment field, and can retransmit it quickly without waiting for the rto timer to time out.

Fast retransmission is followed by fast recovery, and the sender will halve the ssthresh threshold. Considering that the sender receives repeated acknowledgements, the network congestion has been alleviated, so you can set the congestion window cwnd to ssthresh+3. Why add 3, according to the principle of "packet conservation", the number of packets in the network pipeline is constant at the same time, and three duplicate ack are received, indicating that three packets have left the network pipeline.

The change of sender traffic over time is as follows:

The cubic algorithm is used in the Linux kernel, and the basic principle is similar to reno (reno itself is also evolving). One reason for being slow to perceive real networks is that the ssthresh threshold is difficult to predict. Linux4.9 already supports bbr. At present, Taobao network client has optimized the protocol stack based on bbr. This piece is free to write.

On how to achieve file download breakpoint continuation and what are the basic features of TCP to share here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.