Shulou(Shulou.com)06/01 Report--

This article will explain in detail what is the principle of ipvs cluster ip implementation in k8s-service. The content of the article is of high quality, so the editor will share it with you for reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

The load balancing of cluster ip type and node port type service in k8s cluster based on iptable. In essence, when the network packet enters the network namespace in the host host from the network namespace of pod through the linux veth pair device, it DNAT the cluster ip and port of service into the ip and port of pod through a series of NAT conversions of iptable. At the same time, the random module of leverage linux iptable realizes the load balance of pod, and then sends the packet to pod by the routing strategy of host to target pod. Of course, all of this is done in the linux kernel space, regardless of the user space of the application. Here we mainly introduce the implementation principle of cluster ip type service based on ipvs.

To achieve ipvs-based k8s network load balancing, you need the following:

The Linux kernel is higher than 2.4.x. Refer to the following:

Https://en.wikipedia.org/wiki/IP_Virtual_Server

Add-proxy-mode=ipvs to the startup parameters of the kube-proxy network component, which was introduced in previous articles

Install ipvsadm tools, essentially ipvsadm is a user-space program used to operate and manage ipvs

Install the ipvsadm tool:

Yum install ipvsadmwhereis ipvsadm

We are here to introduce service of type cluster ip, so we will display the cluster ip in the k8s cluster based on the application installed in the previous article:

Kubectl get service-all-namespaces

According to previous articles, packets enter host's network namespace from pod's network namespace through the linux veth pair device. Host enables routing and forwarding. The data first enters the PREROUTING chain of iptable. Let's take a look at this chain:

Iptables-nL-t nat

According to the chain, the packet will enter into the targe of KUBE-SERVICES.

Check out the target of KUBE-SERVICES:

Iptables-nL-t nat

According to KUBE-SERVICES target, packets match ipset KUBE-CLUSTER-IP. Ipset is the kernel data structure of linux, which can store some ip and port information. Clusters in ipvs mode reduce the number of entry in iptable by matching ipset in iptable. After matching the ipset here, you enter the target of KUBE-MARK-MASQ.

Check out the ipset of KUBE-CLUSTER-IP:

Ipset list KUBE-CLUSTER-IP

Here we see that there are a total of nine entry in the KUBE-CLUSTER-IP ipset, and they also match the ip and port of the cluster ip type service in the cluster.

According to KUBE-SERVICES target, the items that matches the ipset of KUBE-CLUSTER-IP (that is, the service of type cluster ip) will enter the target of KUBE-MARK-MASQ. Let's look at the target:

We observe that this target is marked with mark for all items.

After the PREROUTING chain and the associated target, the data comes to the INPUT chain, because for the ipvs mode, the kube-ipvs0 network device is created in the network namespace of the host, and all the cluster ip is bound so that the data can enter the INPUT chain.

Ip addr | grep kube

For ipvs load balancer of K8s cluster, its core work is to use NAT mode in INPUT chain (http://www.linuxvirtualserver.org/VS-NAT.html, of course, ipvs not only has NAT mode, but also has more modes. For more information, please refer to www.linuxvirtualserver.org/Documents.html on the official website of LVS project). The network kernel of linux operating system will transfer DNAT to the target ip. Here we take service service-nginx-app as an example. Its cluster ip is 10.254.226.173. Let's see how ipvs does DNAT.

Kubectl describe service service-nginx-app-namespace defaultipvsadm-L

We see that the service service-nginx-app cluster ip is 10.254.226.173 endpoint 80, and the corresponding two are 10.1.86.6 and 10.1.86.7 80. Then use the ipvsadm tool to check that it is indeed ipvs that maps it to two endpoints, and uses the round robin allocation method to assign weights of 1 and 1, that is, to achieve load balancing evenly.

Ipvs does the above DNAT in INPUT chain, and then feeds the data into POSTROUTING chain, and we look at the chain.

Iptables-nL-t nat

Here we find that the data in POSTROUTING chain will enter into the target of KUBE-POSTROUTING.

View KUBE-POSTROUTING target:

Iptables-nL-t nat

Here we find that the packet is camouflaged by MASQUERADE, and what matches is the mark made in KUBE-MARK-MASQ target, that is, the SNAT operation is done with the ip of the network device used in the next-hop routing. So here our packet source ip is the ip of the network device used for the next-hop routing, with a destination ip of 10.1.86.6 or 10.1.86.7 (RR 1:1 load balancer), and then make the next-hop route selection according to the routing table of host network namespace.

The communication methods for cluster ip under ipvs are summarized as follows:

The packet is sent from the pod network namespace and enters the network namespace of the host. The source ip is pod ip, the source port is random port, the destination ip is cluster ip, and the destination port is the specified port.

The packet enters the PREROUTING chain in host network namespace.

Do the mask tag operation through the matching ipset KUBE-CLUSTER-IP in PREROUTING chain.

Create a network device kube-ipvs0 in host network namespace, and bind all cluster ip, so that the destination ip of the packet sent from pod is cluster ip, corresponding to the kube-ipvs0 network device, and the data enters the INPUT chain.

The data is modified by ipvs kernel rules in INPUT chain (rules can be viewed by ipvsadm), DNAT is completed, and the data is then fed directly into POSTROUTING chain. In this case, the source ip is pod ip, the source port is random port, the destination ip is the pod ip selected by the mapping, and the destination port is the port selected by the mapping.

The data is in POSTROUTING chain, and the MASQUERADE SNAT is completed through KUBE-POSTROUTING target. In this case, the source ip is the ip of the network device used in the next-hop routing, the source port is the random port, the destination ip is the pod ip selected by the mapping, and the destination port is the port selected by the mapping.

The packet makes the next-hop route selection according to the routing table of the host network namespace.

What is the principle of ipvs cluster ip implementation in k8s-service is shared here, I hope the above content can be of some help to you, you can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.