Shulou(Shulou.com)06/01 Report--

Introduction: there are two schemes for Aliyun K8S cluster network: one is flannel scheme, and the other is terway scheme based on calico and elastic network card eni. Terway is similar to flannel, except that terway supports Pod ENI and NetworkPolicy functions. In this paper, based on the current version 1.12.6, the author takes flannel as an example to deeply analyze the implementation method of Ali Yun K8S cluster network. In general, after the configuration of Aliyun K8S cluster network is completed, as shown in the following figure: it includes cluster CIDR, VPC routing table, node network, node podCIDR, virtual bridge cni0 on node, veth connecting Pod and bridge, and so on.

Similar diagrams may have been seen in many articles, but because the relevant configuration is too complex, it is difficult to understand. Here we can look at the logic behind these configurations. Basically, we can understand these configurations in three situations: cluster configuration, node configuration, and Pod configuration. What corresponds to these three cases is actually three divisions of the cluster network IP segment: first, the cluster CIDR, then the podCIDR (that is, the subnetwork segment of the cluster CIDR) is assigned to each node, and finally, each Pod is assigned its own IP in the podCIDR.

In the initial stage of building a cluster network, the cluster is created based on cloud resources VPC and ECS. After creating VPC and ECS, we can basically get the resource configuration shown in the figure below. We get a VPC, the network segment of this VPC is 192.168.0.0 VPC 16, and we get several ECS, which are assigned to IP addresses from the VPC segment.

In the cluster phase, on the basis of the above initial resources, we use the cluster creation console to get the cluster CIDR. This value is passed as a parameter to the cluster node provision script and is passed to the cluster node configuration tool kubeadm by the script. Kubeadm finally writes this parameter to the yaml file kube-controller-manager.yaml of the static Pod of the cluster controller.

The cluster controller has this parameter, and when the node kubelet registers the node to the cluster, the cluster controller will divide a subnet for each registered node, that is, assign podCIDR to each node. As shown in the figure above, the subnet of Node B is 172.16.8.1Comp25, while the subnet of Node An is 172.16.0.128Comp25. This configuration is recorded in the podCIDR data item of the cluster node. The node phase goes through the above cluster phase, and K8S has a cluster CIDR and a podCIDR divided for each node. On this basis, the cluster will send flanneld to each stage, and further build a network framework on the node that can be used by Pod. There are two main operations here: the first is that the cluster configures routing table entries to VPC through Cloud Controller Manager. There is one routing table entry for each node, which means that if the destination address received by the VPC route is the IP address of a node's podCIDR, then the route forwards the network packet to the corresponding ECS; the second is to create a virtual bridge cni0 and cni0-related routes. The purpose of these configurations is that the network packets coming from outside the phase, if the destination IP is podCIDR, will be forwarded by the node to the cni0 virtual LAN. Note: in practice, the creation of cni0 is created by flannal cni in the next section when the first Pod using the Pod network is scheduled to the node, but logically, cni0 belongs to the node network, not the Pod network, so it is described here.

The Pod phase is in the previous three stages, and the cluster has actually set up a trunk road for network communication between Pod. At this time, if the cluster dispatches a Pod to the node, kubelet creates a network namespace and veth device for the Pod itself through flannel cni, then adds one of the veth devices to the cni0 virtual bridge and configures the IP address for the veth device in the Pod. In this way, the Pod is connected to the main road of network communication. It is important to emphasize that the flanneld in the previous section and the flannel cni in this section are completely two components. Flanneld is a pod sent by daemonset to each node, and its function is to build a network (trunk road), while flannel cni is a cni plug-in installed through the rpm package kubernetes-cni when a node is created, which is called by kubelet to create a network (branch) for a specific pod. Understanding the difference between the two helps us understand the purpose of flanneld and flannel cni-related configuration files. For example, / run/flannel/subnet.env, created by flanneld, is an environment variable file that provides input for flannel cni; another example is / etc/cni/net.d/10-flannel.conf, which is also a flanneld pod (script install-cni in pod, to be exact) copied from pod to the node directory for the subnet configuration file used by flannel cni.

The communication above completes the construction of Pod network environment. Based on the above network environment, Pod can complete four kinds of communication: local communication; Pod communication with the same node; cross-node Pod communication; and entity communication outside the Pod and Pod network.

The local communication refers to the communication between different containers within the Pod. Because Pod intranet containers share a network protocol stack, the communication between them can be done through loopback devices. The communication between the Pod of the same node is the communication within the cni0 virtual bridge, which is equivalent to the internal device communication of a two-layer local area network. Cross-node Pod communication is slightly more complex, but also intuitive. The sender packet passes through the gateway of the cni0 bridge, flows to the node, and then is sent to the VPC route through the node eth0. There will be no packet operations here. When a VPC route receives a packet, it queries the routing table, confirms the destination of the packet, and sends the packet to the corresponding ECS node. After entering the node, because flanneld creates a route to cni0 on the node, the packet is sent to the destination cni0 LAN and then to the destination Pod. Finally, when Pod communicates with entities of a non-Pod network, it needs to SNAT through the iptables rule on the node, which is the configuration of flanneld according to the command line-ip-masq option. To sum up the above is the construction and communication principle of Ali Yun K8S cluster network. We mainly analyze the K8S cluster network from the perspectives of network construction and communication. Network construction includes initial phase, cluster phase, node phase and Pod phase. This classification helps us to understand these complex configurations. With the understanding of each configuration, the principle of cluster communication is easier to understand. Author: Shengdong Aliyun after-sales technical expert original link this article is the original content of Yunqi community and may not be reproduced without permission.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.