Shulou(Shulou.com)06/01 Report--

This article introduces how to understand the network principle analysis in Kubernetes. The content is very detailed. Interested friends can use it for reference. I hope it will be helpful to you.

01 overlay network

Overlay overlay network is a technology that wraps TCP data in another "network packet"incoming" route for forwarding and communication. Overlay contacts are not required by default, but they are "common" in specific scenarios. For example, when we don't have enough IP space, or when we need some additional management features that Overlay provides. The usual scenario is that when there is a limit on the number of routes that can be handled by the cloud provider's routing table, for example, a maximum of 50 routes in the AWS routing table will not affect the network performance. So if we have more than 50 Kubernetes nodes, the AWS routing table will not be enough. In this case, making "Overlay" will help us.

In essence, Overlay is to re-encapsulate the layer package in the package on the local network across the node. You may not want to "Overlay" because it brings latency and complexity overhead caused by encapsulating and de-encapsulating all packets. Usually this is unnecessary, so we should use it only when we know why we need it.

To understand the flow of traffic in Overlay networks, let's take Flannel as an example, which is an open source item of CoreOS. Flannel provides virtual coverings to the container by assigning each host in the form of a packet. Based on Linux TUN/TAP, it enables UDP to encapsulate IP packets to create overlay networks, and maintains the allocation of overlay networks with the help of etcd.

Kubernetes Node with route table (cross-node Pod-to-Pod communication over Flannel Overlay network)

We noticed that it was similar to the facility we saw before, except that a virtual Ethernet device, called flannel0, was added to root netns. It is a kind of implementation of virtual extension Virtual Extensible LAN (VXLAN), but on Linux, it is just another network connection.

The flow of packets from pod1 to pod4 (at different nodes) is similar to the following:

1) it is left by the eth0 administrator of the netns in pod1 and enters the root netns through vethxxx.

2) it is then passed to cbr0, and cbr0 finds the tagged address by sending an ARP request.

3) data encapsulation

3a. Because there is no Pod on this node that has the IP address of pod4, the bridge sends the packet to flannel0 because the flannel0 on the node's routing table is assigned to the Pod segment.

3b. Flanneld daemon communicates with Kubernetes apiserver or the underlying etcd. It knows all the Pod IP and knows which node they are on. So Flannel creates a mapping between Pod IP and Node IP (in the subscriber space). Flannel0 fetches the packet and encapsulates it with a UDP packet. The source of the UDP packet header and the IP of the UDP packet are changed to the IP of the corresponding node, and then the new packet is sent to a specific VXLAN terminal (usually 8472).

Packet-in-packet encapsulation (notice the packet is encapsulated from 3c to 6b in previous diagram)

Although this mapping occurs in user space, real encapsulation and data flow occur in kernel space, so it is still very fast.

3c. The encapsulated packet is sent over eth0 because it involves routed traffic between nodes.

4. The packet leaves the node with the IP information of the node as the address of the source and sink.

5. The routing table of the cloud provider already knows how to send messages between nodes, so the newspaper is "sent" to the labeled address node2.

6. Data unpacking

6a. The packet arrives at the eth0 card of node2, and because the header end is a specific VXLAN terminal, the kernel sends the message to the

Flannel0 .

6b. Flannel0 unencapsulates the message and sends it to the root namespace. From this point on, the path of the report is caused by the "Overlay" we saw earlier in Part1.

6c. Because IP forwarding is turned on, the kernel forwards the message to cbr0 according to the routing table.

7. The IP bridge gets the packet, sends an ARP request, and finds that the token IP belongs to vethyyy.

8. The packet crosses the pipe pair to reach the pod4

This is the "action" form of Overlay in Kubernetes, although there are slight differences between different implementations. There is a common misunderstanding that when we Kubernetes, we have to make Overlay. The truth is, it all depends on the specific scenario. So make sure you use it only in situations where you really need it.

02 dynamic cluster

Because of the ever-changing nature of Kubernetes (more commonly referred to as distributed systems), its Pod (and Pod's IP) is always changing. The reason for the change can be rolling upgrades and extensions for unpredictable Pod or node crashes. This makes it impossible for Pod IP to communicate directly. Let's take a look at the Kubernetes Service, which is a virtual IP, accompanied by the Endpoint group Pod IP (identified by the tag selector). They act as virtual load balancers, their IP remains the same, and the back-end Pod IP may be constantly changing.

Label selector in Kubernetes service object

The implementation of the entire virtual IP is actually a virtual group iptables (the latest version can choose to make the virtual IPVS, but this is another discussion) rules, managed by the Kubernetes component kube-proxy. The name is now actually misleading. It was indeed a proxy before v 1.0, and because its implementation was a constant replication between kernel space and subscriber space, it became resource-intensive and slow. For now, it's just a controller, just like many other controllers in Kubernetes, which watch api serverendpoint changes and update iptables rules accordingly.

Iptables DNAT

With these iptables rules, whenever a packet is sent to Service IP, it goes into a DNAT (DNAT= label address translation) operation, which means that the label IP changes from Service IP to one of the Endpoint-Pod IP-randomly selected by iptables. This ensures that the load is evenly distributed in the back-end Pod.

5-tuple bars in the conntrack table

When the DNAT is posted, this information is stored in the conntrack-- the Linux connection tracking table (iptables Rule 5 tuples are translated and stored: protocol, srcIP, srcPort, dstIP, dstPort). So when the request comes back, it can un-DNAT, which means changing the source IP from Pod IP to Service IP. In this way, the client does not care about how the background handles the packet flow.

So by making Kubernetes Service, we can make sure that the same end does not cause any conflicts (because we can remap the end to Endpoint). This makes service discovery easy. We can make "internal DNS" and hard-code the service host name.

We can even use the service host and client environment variables provided by Kubernetes to complete service discovery.

Experts suggest that by adopting the "DNS" method, you can save unnecessary DNS calls, but due to the limitation of the creation order of the environment variables (the environment variables do not contain the services created later), it is recommended to use the "DNS" service name resolution.

2.1 outbound traffic

The Kubernetes Service we are talking about is done within a cluster. However, in most real-world situations, the program needs to access some external api/website. Typically, a node can have both private IP and public IP. For connected IP access, there is some kind of 1:1 NAT for these public and private NAT, especially in the cloud. For normal communication from a node to some external IP, the source IP changes from the dedicated IP of the node to the public IP of its outbound packet, while the response packet of the sink station is just the opposite. However, when the Pod issues a connection to the external IP, the source IP is Pod IP, and the cloud provider's NAT mechanism is not aware of the IP. Therefore, it will drop packets with a source IP other than the node IP.

So you may also be right, we will make more iptables! These rules are also added by kube-proxy to enforce SNAT (source network address translation), or IP MASQUERADE (IP camouflage). It tells the kernel to connect the IP sent out by this packet instead of the source Pod IP while retaining conntrack bars to facilitate anti-SNAT operations.

2.2 inbound traffic

It's good to cut it for you before you get there. Pod can talk to each other or visit the Internet. But we still lack the key part-providing services for customer request traffic. Before the cut-off, there are two main ways to do this:

NodePort / Cloud load balancer (L4-IP and end-end balancer)

Setting the service type to NodePort allocates nodePort with a range of 30000-33000d to the service by default. Even if there is no Pod running on a specific node, this nodePort opens on each node. The Pod traffic on this NodePort will once again cause the iptables to be sent to one of the Pod (the Pod may even be on another node! ).

The LoadBalancer service type in the cloud environment will create a cloud load balancer (such as ELB) before all nodes, hitting the same nodePort.

Ingress (L7-HTTP / TCP)

Many different appliances, such as Nginx, Traefik, HAProxy, etc., retain the http hostname / path and the mapping of each slave backend. Usually, this is a traffic threshold based on load balancer and nodePort. Its advantage is that we can have one load balancer to handle the traffic of all services, without the need for multiple nodePort and load balancers.

2.3 website Policy

Think of it as Pod's security group / ACL. NetworkPolicy rules allow / deny traffic across the Pod. The exact implementation depends on the network layer / CNI, but "most of them only make" iptables.

That's it for the former. In the previous part, we studied the basis of Kubernetes collaterals and the principle of overlay collaterals. Now we know how Service abstractions act within dynamic clusters and make service discovery often easy. We also introduce the principle of outbound and outbound traffic and how the network strategy acts on the security in the cluster.

About Kubernetes in the network principle analysis how to understand to share here, I hope the above content can have some help to everyone, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.