Shulou(Shulou.com)11/24 Report--

Usually, we want to know that between our own machine and the target machine, the network is not connected, and the Ping command is generally executed.

Generally, for a good network, you can see that its corresponding loss packet loss rate is 0%, which is the so-called ping. If you see a packet loss rate of 100%, that is, Ping does not work.

Ping OK

Ping is not working.

So the question comes, assuming I can ping a machine, then if I switch to TCP protocol to send data to the target machine, will it be able to pass?

Or to put it another way, are Ping and tcp protocols taking the same network path?

At this time, the first reaction is not necessarily, because after Ping, a router in the intermediate link may hang up (power off), and then use TCP to connect will take another path.

That's true. But suppose nothing changes on the intermediate link?

I'm gonna get right to the answer.

Not necessarily, the network path taken may still be different.

Let's talk about why today.

Difference Between Ping and TCP Messages

We know that networks are layered, with protocols corresponding to each layer.

Analysis of Message Body Changes Corresponding to Five-Layer Network Protocols

This network layer was like building blocks. The upper protocols were built based on the lower protocols.

Whether it is Ping (using ICMP protocol) or tcp is essentially based on the network layer IP protocol packets, and to the physical layer, are binary 01 string, are sent out through the network card.

If the network environment has not changed and the destination is the same, then logically speaking, the network path they take should be the same. Under what circumstances will they be different?

Let's start with routing.

Network paths are in our imagination when we want to transfer data between two machines. A connection is established between the machine itself and the target machine, like a pipeline, from one end of the data to the other. This pipeline is actually an abstract concept that we abstract for ease of understanding.

In fact, after we send the packet from the local NIC, it will pass through various routers (or switches) before reaching the destination machine.

These routers are numerous in number and can be interconnected with each other. After being connected, they are like a large network, so it can be said that "network" is a very image.

A network of routers

Considering the functions of switches, routers basically support them, so we only discuss routers here.

So now the question comes, after the router receives the data, how does it know which path to take and which router to send?

What determines the path? In a network as large as the one above, any router can take any path to send data to another router.

However, the distance between routes and routes, bandwidth and so on may vary.

It is important to know which path is best between two points.

So the problem becomes this graph structure. Each edge has a cost or weight, and the shortest distance between any two points on it.

Routers and Dijkstra

At this time, everyone must have been unable to suppress their memories.

I know this question well. This is the Dijkstra algorithm that I brushed in college. Chrysanthemum Factory's OJ written test set also often appeared, now finally understand why their family's written test questions seem to have more picture questions than other big factories, because chrysanthemum factory is engaged in communication, do router veteran players.

Generation of Routing Table

Based on the Dijkstra algorithm, a new protocol, OSPF (Open Shortest Path First), was encapsulated.

With OSPF, a router gets the shortest distance between itself and other points in the network graph, so it knows which best path a packet should take to get to a certain point.

Put this information into a table, which is often referred to as a routing table.

The routing table records which ports need to be used to get to which IP, and the cost (metric) of taking this path.

It can be seen by the route command.

route table

Routing tables determine packet paths

The destination IP address is added at the network layer as the packet is sent.

The router matches the IP address with the routing table.

The routing table then tells the router what messages should be forwarded to what ports.

Take an example.

Forward data through routing tables

Suppose A sends a message to D. That is, 192.168.0.105/24 should send a message to 192.168.1.11/24.

Then A will send the message to the router.

The router knows the destination IP192.168.1.11/24, matches the routing table, finds 192.168.1.0/24, on port e2, then sends the message from port e2,(possibly through the switch) and finally sends the message to the destination machine.

Of course, if you can't find it in the routing table, then call the default gateway, that is, from the e1 port, sent to IP192.0.2.1. This router's routing table doesn't know where to go. Maybe other routers do.

Matching rules for routing tables

In the example above, it matches only one entry in the routing table, so it can only be it.

But all roads lead to Rome. There must actually be many ways to get there.

If there are a lot of matches in the routing table, what will you choose?

If more than one routing entry can reach the destination, the one with the longer matching length is preferred. For example, if the destination is still 192.168.1.11, it is found that 192.168.1.0/24 and 192.168.0.0/16 in the routing table can be matched, but obviously the matching length of the former is longer, so the forwarding port corresponding to 192.168.1.0/24 will be used in the end.

But what if both entries match the same length?

Then it depends on what protocol generates this routing table entry, and the higher the priority, the smaller the so-called Administrative Distance (AD). For example, manually configured static routes are preferred over entries dynamically learned by OSPF.

If it is still the same, look at the metrics, which is actually the path cost. The smaller the cost, the easier it is to be selected.

There are many routes that routers can choose, but in principle, the optimal one is only "one," so up to here, we can think that for the same destination, Ping and TCP take the same path.

But...

What if the path costs are the same? That means there are multiple optimal paths.

Then use both.

This is called ECMP (Equal Cost Multipath).

Traceroute can be used to see if there are equivalent multipath cases.

You can see that there are several IPs in a few lines in the middle, which means that several target machines can be selected at the same time in this hop, indicating that this path supports ECMP.

What is ECMP for?

With equivalent multipath, we can increase link bandwidth.

Take an example.

Without ECMP, only one path can be selected

From point A to point B, if the two paths cost different, the bandwidth is 1 gigabit. The packet must have chosen the low-cost path. If this path fails, it will take the following path. But anyway, at the same time, only one path was used. The other one is a waste if it is idle. Is there any way to make use of it?

Yes, set the cost of their two paths to the same, then they become equivalent routes, and then the router in the middle turns on the ECMP feature, you can use both links at the same time. bandwidth went from 1 gigabit to 2 gigabytes. The data can choose between two paths.

ECMP allows two links to be used simultaneously

But that raises another question. Increased packet disordering.

It turned out that I only used one network path, and the data was sent out in turn, and if there was no accident, it was also arrived in turn.

Now two packets take two paths, and the first packet may arrive later. That's out of order.

Then the question arises again.

What's wrong with disorder?

For our most commonly used TCP protocol, it is a reliable network protocol. The reliability mentioned here is not only to ensure that the data can be sent to the destination, but also to ensure that the data order is the same as the original sender.

The implementation is also simple, TCP numbers each packet (segment). After the data arrives at the receiving end, it is found that it is an out-of-order packet according to the packet number, and it will be thrown into an out-of-order queue to sort the packets. If the previous data packet has not arrived, even if the latter data packet arrives first, it has to wait in the out-of-order queue until it can be obtained by the upper layer.

For example, the sender sends three packets, numbered 1, 2, and 3, assuming they arrive first at transport layers 2 and 3, but not yet at 1. At this time, the application layer cannot get the data packets 2 and 3. It must wait for 1 to come before the application layer can get these three packets at once. Because these three packets may originally represent a complete message, if 1 is missing, then the message is incomplete, and it is meaningless for the application layer to get it.

Such a phenomenon, due to the loss of data in front of the data can not be given to the application layer in time, is what we often say TCP queue head blocking.

Out-of-order queues waiting for packets to arrive

When out-of-order occurs, 2 and 3 need to stay in the out-of-order queue, and the practical use of out-of-order queue is also the memory of the receive buffer, which is limited in size. The size of the receive buffer can be seen by following the command.

#View receive buffer $ sysctl net.ipv4.tcp_rmemnet.ipv4.tcp_rmem = 4096(min) 87380(default) 6291456(max)#The more the buffer dynamically adjusts the disorder between min and max, the more the memory of the receive buffer will be occupied, the corresponding receive window will become smaller, the less data can be received normally, and the network throughput will be worse, that is, the performance will be worse.

Therefore, we need to make sure that all TCP packets under the same TCP connection take the same path, so as to avoid packet loss to the greatest extent.

Path Selection Strategy of ECMP

ECMP was originally turned on to improve performance, but now it increases the disorder and reduces TCP transmission performance.

How could he tolerate this?

In order to solve this problem, we need to have a reasonable path selection strategy. In order to avoid packet disordering within the same connection, we need to ensure that packets within the same connection all follow the same path.

That's easy. We can locate a unique connection by five-tuple information about the connection (IP and port of sender, IP and port of receiver, and communication protocol).

five-tuple

Then generate hash keys for the five-tuple information, let the data of the same hash key take the same path, and the problem is solved perfectly.

quintuple mapping to hash key

Select ECMP path according to quintuple

TCP and Ping take the same network path? Now let's go back to the question at the beginning of this article.

For the same sender and receiver, do TCP and Ping follow the same network path?

It doesn't have to be the same, because one of the items in the quintuple is a communication protocol. Ping uses ICMP protocol, which is different from TCP protocol, and Ping does not need to use ports, so the five tuples are different, the hash keys generated are different, and the paths selected through ECMP may also be different.

Five-tuple difference between TCP and Ping

The same TCP protocol is used, is the network path for data packets the same or the same sender and receiver, the same TCP protocol, is the network path for different TCP connections the same?

Like the above problem, it is actually a five-tuple problem. It is also a TCP protocol. For the same sender and receiver, their IP and the port of the receiver must be the same, but the port of the sender can change at any time. Therefore, the path taken through ECMP may also be different.

Five-tuple differences for different TCP connections

But here's the problem.

I know. What's the point? I do business development, and I don't have permission to set up network routes.

Using this knowledge point to troubleshoot problems is definitely not a useless knowledge point for business development.

If one day you find that you can Ping the target machine, but use TCP to connect, but occasionally can not connect to the target machine. Moreover, the machines at both ends were quite idle, and there were no bottlenecks in performance. It was desperate.

You can think about whether ECMP is used in the network and one of the links has a problem.

Ping succeeds but some TCP connections fail

The screening method is also very simple.

You know the IP of your machine and the IP and port number of the destination machine, and you know that you are using TCP connections.

As long as you print the error message when reporting the error, you know the port number of the sending end.

So you know what a quintuple is.

The next step is to specify the port number of the sending end to re-initiate TCP requests, the same five-tuple, take the same path, it is reasonable to say that if there is a problem with the link, it will definitely repeat.

If you don't want to change your own code, you can specify the client port with the nc command to see if the TCP connection can be established properly.

nc -p 6666 baidu.com 80-p 6666 specifies that the client port from which the request originated is 6666, followed by the domain name of the connection and port 80.

TCP connection successfully established through nc

Assuming that five-tuple connections with port 6666 always fail, but 6667 or other ports succeed, you can take this information to find colleagues in charge of the network.

Summarizing routers can generate routing tables through OSPF protocol, match IP addresses in packets with routing tables, and select the best path for forwarding.

When none of the routing tables match, the default gateway is taken. When there are multiple matches, the match length will be looked at first. If it is the same, the management distance will be looked at. If it is the same, the path cost will be seen. If even the path costs are the same, then the equivalent path. If ECMP is enabled for the route, it can use these paths for transmission at the same time.

ECMP can improve the link bandwidth, and at the same time use quintuple as hash key for path selection, ensuring that the data packets of the same connection take the same path, reducing the disorder.

The traceroute command can be used to see if ECMP is available on the link.

In the network link with ECMP turned on, TCP and Ping commands may take different paths, even the same TCP, between different connections, take different paths, so there is a problem of good or bad connection, it is really desperate, you can consider whether it is related to ECMP.

Of course, when encountering problems, doubt yourself more, and believe that most of the time it has nothing to do with ECMP.

References "Network Troubleshooting Case Lesson"-Geek Time

This article comes from Weixin Official Accounts: Xiaobai debug (ID: xiaobaidebug), author: Xiaobai

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.