Shulou(Shulou.com)05/31 Report--

In this article, the editor introduces "what is the principle of Kubernetes network" in detail, the content is detailed, the steps are clear, and the details are handled properly. I hope this article "what is the principle of Kubernetes network" can help you solve your doubts.

Principles and challenges of Part.1 Kubernetes network 1. Kubernetes Pod design

Pod is the basic scheduling unit of Kubernetes. We can think of Pod as an extension of containers, a Pod is also an isolator, and Pod contains a set of shared containers.

The container in each Pod consists of a special Pause container and one or more closely related business containers. The Pause container is the root container of Pod, and the corresponding image is part of the Kubernetes platform, and its state represents the state of the entire container group. Containers in the same Pod can communicate with each other only through localhost.

Each Pod is assigned a unique (IP address in the cluster, called Pod IP) by the Kubernetes network component, which allows services in different Pod to use the same port (ports in the same Pod can only be occupied by one service), avoiding port conflicts.

two。 Challenges

Pod's IP is within the Kubernetes cluster, virtual and local. This brings a problem. There is no problem in accessing Pod IP within the cluster, but in the actual project development, there will inevitably be applications in the real network environment that need to access the containers in the Kubernetes cluster, which requires us to do some extra work.

Combined with the process of network evolution based on K8S container in our development environment, this paper introduces several solutions to access Pod IP directly when Pod IP is virtual or real IP.

The first stage of Part.2 Kubernetes-based Container Network Evolution: Kubernetes + Flannel

At first, our network design only served the large transportation business. In the early days, the Java application of Daotong was deployed on the physical machine, and the underlying infrastructure related to containers within the team was almost zero. In order to achieve a more stable containerization landing, we did not directly migrate all the services to K8S, but experienced a mixed run.

In this process, it is inevitable that the Java application on the physical machine accesses the Java container in K8S (a registry). At that time, the network architecture we chose was Flannel VXLAN + Kube-proxy, mainly considering the simplicity of Flannel.

Flannel is a very simple multi-node three-layer network scheme designed for K8S, which is mainly used to solve the problem of cross-host communication of containers. It is designed to replan the rules for the use of IP addresses for all nodes in the cluster, so that containers on different nodes can obtain IP addresses that "belong to the same intranet" and "do not repeat", and containers belonging to different nodes can communicate directly through the intranet IP.

The principle of Flannel is to assign a subnet to each host, and the container assigns IP from this subnet. These IP can be routed between host, and the containers can communicate across hosts without NAT and port mapping. Each subnet is divided from a larger IP pool, and Flannel runs a daemon flanneld on each host, which is responsible for allocating subnet from the large pool to share information among hosts. Flannel uses ETCD to store network configuration, assigned subnet, IP of host, and other information.

There are three ways of communication between nodes of Flannel:

VXLAN: default configuration, using kernel-level VXLAN to encapsulate packets transferred between host

Host-gw: layer 2 network configuration. Cloud environment is not supported. Route entries to other host subnet are created directly in the routing table of host.

UDP: commonly used for debug

We deploy Flannel on all business physical machines, and start the Flanneld service to join the K8S virtual network, so that when the service on the physical machine and the container service in K8S call each other, they can pass through the virtual network of Flannel+Kube-proxy. The overall structure is as follows:

We deployed VPN Server as nodeport under the middleware space of the cluster, and assigned 10.140 of the network segment to the client.

When the client dials VPN through 172.18.12.21Vol 30030, the network between the client and the VPN Server is connected.

Because the VPN Server itself is in the cluster virtual network environment, the IP of other containers is visible to the vpn server, so after dialing VPN, the VPN client can directly access the Pod in the cluster

The network architecture diagram between the modified development environment and the K8S cluster of the computer room is as follows:

By setting up OpenVPN in K8S cluster, we have temporarily solved the problem of RPC communication between office area and K8S cluster in computer room. The advantages and disadvantages of this scheme are as follows:

Advantages:

Fast implementation

The engineering quantity is small.

Network isolation, certificate verification is more secure

Deficiency:

The operation is a little tedious, for example, you need to apply for a certificate and install the client software; you need to open OpenVPN before each use.

It is an intermediate solution that does not solve the problem in essence.

The third stage: Kubernetes CNI based on MAC-VLAN

In order to better support the micro-service transformation of Java business and avoid repeating the wheel, we have built a unified Java basic platform, and the previous network solution also provides services for more departments.

With more and more business and personnel of the service, the inconvenience and problems caused by manual operation are becoming more and more obvious, so we decided to transform the K8S network again. From previous experience, we feel that if the virtual network of K8S is not removed, the IP of CCS cannot be reached directly from the real network outside the cluster.

In order to quickly meet the needs of K8S network functions in different business scenarios, we began to study CNI in depth.

About CNI

CNI (Conteinre Network Interface) aims to provide a unified network standard for the container platform, which is developed by Google and CoreOS. It is not a complete solution or program code, but a container network interface protocol developed on the basis of RKT after considering flexibility, expansibility, IP allocation, multiple network cards and other factors.

The network classification and mainstream implementation tools of CNI mainly include:

Category 1: flat contact with the host (2 or 3 layers). Plug-ins mainly include Bridge, MAC-VLAN, IP-VLAN, Calico BGP, Flannel host-GW, etc.

The first category: virtual network of SDN technology, the main plug-ins are: Flannel vxlan, Calico ipip, Weave, etc.

MAC-VLAN and its problems

Through comparison and testing, taking into account the current needs, we finally decided to based on the MAC-VLAN solution with low network transformation, low operation and maintenance cost and complexity:

MAC-VLAN has the feature of Linux Kernal to configure a virtualized interface for a physical network interface (parent). The virtualized interface and the parent network interface have different mac addresses, but when mac packets are sent to the corresponding virtualized interface on the parent interface, they will be distributed to the corresponding virtualized interface, which is a bit like bridging the virtualized interface with the parent interface, so that the virtualized network interface can access each other after IP and routing are configured.

In the MAC-VLAN scenario, K8S container access may encounter some problems, for example, after configuring MAC-VLAN, the container cannot access the IP of the parent interface.

Through research, it is found that there are two solutions:

1. Virtual network card: open the network card hybrid mode, and aggregate and bind the virtual network card to the real network card of the host by virtualizing a virtual network card on the host computer.

2. PTP scheme: similar to the simplified version of Bridge, but the network configuration is more complex, and some configurations are found to be of little use during self-test. The main difference from Bridge is that PTP does not use bridges, but directly uses vethpair+ routing configurations.

By comparing the implementability of the two schemes, we finally chose the first one, but after testing in the virtual machine environment, it is found that the host is occasionally not connected with the local container. Students who adopt the first scheme are advised not to use the virtual machine environment as far as possible (it is suspected that it is still a problem of mixed setting of network cards).

The Network Reform of "governing by Division"

KubeDNS, the core component of K8S, defaults to accessing the first IP; of the ClusterIP segment at startup. If you continue to use the original Nginx-ingress, then the Nginx-ingress container also uses ClusterIP to connect to the KubeDNS at startup. Using MAC-VLAN to assign real network IP to KubeDNS and Nginx-ingress, they cannot connect to ClusterIP, so KubeDNS and Nginx-ingress containers cannot use MAC-VLAN solution, otherwise the domain name access capability of CCS will be lost.

After comprehensive consideration, we finally adopted the measure of "partition and governance", dispatching different types of containers to different areas and using different network solutions, which are roughly illustrated as follows:

When using the MAC-VLAN scheme, there are two allocation schemes for container IP: DHCP and host-local. Considering that the company does not have a unified DHCP and IPAM server to assign IP to the container, and there are only a small number of machines in the development environment, we use manual participation in the network IP segment allocation of each node.

To sum up, the network architecture diagram after this transformation is roughly as follows:

Effect.

You can see the comparison with the architecture diagram of the first network transformation:

Virtual network components such as Kube-proxy and Flannel have been abandoned on Host 1 and Host 2.

Business container nodes such as host 1 and host 2 directly use the company's public DNS facilities

After the local consumer service in the office area gets the IP of provider in the registry, it can be directly connected to the consumer, and vice versa.

K8S cluster is divided into virtual network area (running K8S cluster management components) and real network area (running business container).

The advantages and disadvantages of this transformation are summarized as follows:

Advantages:

Remote debug

The RPC,TCP communication between the office network and the services in the cluster is connected in the layer 2 network.

Network latency is greatly reduced

Support multi-computer room disaster recovery deployment

Disadvantages:

Large amount of engineering

It costs a lot of real IP addresses.

When the cluster size is very large, there is a risk of ARP broadcast storm and switch MAC table exceeding the limit.

After reading this, the article "what is the principle of Kubernetes network" has been introduced. If you want to master the knowledge points of this article, you still need to practice and use it yourself to understand it. If you want to know more about related articles, welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.