Shulou(Shulou.com)05/31 Report--

This article will explain in detail how to analyze the container network in Kubernetes. The content of the article is of high quality, so the editor shares it for you as a reference. I hope you will have some understanding of the relevant knowledge after reading this article.

| | Preface |

With the rise of cloud computing, the competition between major platforms has also come to an end. Kubernetes, as a rising star, has become the de facto PaaS platform standard, and the network is the most complex part of the cloud computing environment, which is always elusive. This paper focuses on the problem of network communication between Pod on the same node (work node) in the Kubernetes environment, regardless of the cross-node network communication.

| | Network NamespaceNamespace |

When it comes to containers, we have to mention the core underlying technology of containers, an isolation mechanism provided by Namespace,Namespace for the Linux kernel, which was first introduced into Linux 2.4.19 in 2002, and only Mount Namespace is used for file system isolation. So far, Linux has provided a total of 7 kinds of Namespace (since Linux 4.6). Each process running in the system is associated with a Namespace, and the process can only see and use resources under that Namespace. Namespace can be simply understood as a "cover-up" for processes implemented by the operating system, for example, through UTS Namespace, processes running on the same machine can see different Hostname. It is because of Namespace's groundbreaking isolation that the implementation of the container is possible, and our software can really implement "build once, running everywhere".

Network Namespace

Network Namespace was not introduced until Linux 2.6.24 and was not completed until Linux 2.6.29. Network Namespace actually implements the virtualization of the network stack, and when it is created, there is only one loopback network interface lo by default. Each network interface (whether physical interface or virtual interface) can only exist in one Network Namespace, but can be switched between different Network Namespace. Each Network Namespace has its own independent IP address, routing table, firewall, socket list and other network-related resources. When you delete a Network Namespace, its internal network resources (network interface, etc.) will also be deleted, while the physical interface will be switched back to the previous Network Namespace.

Container and Pod

In the definition of Kubernetes, Pod is a group of inseparable containers and shares the same Network Namespace, so there is no problem of network communication between containers in the same Pod. For containers in the same Pod, network communication can be carried out with other containers through Localhost.

So the problem of how two Pod on the same node communicate on the network can be transformed into how two containers on the same node communicate on the network.

Namespace practical operation

Before answering the above question about Network Namespace network communication, let's do some simple command-line operations to have a perceptual understanding of Namespace. The experimental environment is as follows:

You can view all the Namespace on the host by using the command lsns (note that you need to use the root user to execute it, otherwise there may be some situations that Namespace can't see):

Lsns outputs all visible Namespace by default, briefly explaining the meaning of each output column of the lsns command:

There is another command, ip netns, which is strongly related to Network Namespace, which is mainly used for the management of persistence namespaces, including the creation, deletion, and configuration of Network Namespace. When creating a Network Namespace, the ip netns command creates a mount point of the bind mount under the / var/run/netns directory by default, thus achieving the purpose of persisting the Network Namespace, that is, allowing the namespace to be preserved even if there are no processes in the namespace. Due to the lack of this step in Docker, students who have played with Docker will find that after creating a container through Docker, they cannot view the relevant Network Namespace on the host through ip netns (I'll talk about how to see it later, just a little bit).

Network Namespace-related operation commands:

Ip netns add

# add network namespaceip netns list # View Network Namespaceip netns delete

# Delete Network Namespaceip netns exec

# go to Network Namespace and execute commands

Create a Network Namespace named netA:

View the created Network Namespace:

You can see that there is only one loopback network interface lo in the Network Namespace netA, and there is a separate routing table (empty).

The host (root network namespace) has network interfaces eth0 (10.10.88.170) and eth2 (172.16.130.164). At this time, you can directly ping IP 172.16.130.164.

Try to add the eth0 interface in root network namespace to network namespce netA:

Ip link set dev eth0 netns netA

After adding the network interface eth0 (10.10.88.170) on the host to the network namespace netA:

1. The eth0 network interface can not be seen on the host (the network interface can only be on one Network Namespace at a time)

2. Cannot ping eth2 in root namespace in netA network namespace (network isolation)

From the above we only know the isolation of Network Namespace, but still can not achieve the desired result, that is, let two containers or two different Network Namespace communicate over the network. In real-life scenarios, when we want to connect the local area network of two branches of the same group, we have three solutions: the first is to connect directly to the public network, which is relatively casual to the data, but there are network security problems. The second kind is not short of money, directly pull a special line to connect the network of the two branches, so that although thousands of miles apart, but can still be in a network. The other is a VPN connection that takes into account both network security set and cost performance, but there are performance problems. Obviously, either solution requires a "wire" to connect the two ends, whether it's a virtual VPN or a physical private line.

| | vEth (Virtual Ethernet Device) |

It is mentioned earlier that the container is isolated from the network through Network Namespace, but because of the isolation of Network Namespace, two different Network Namespace are unable to communicate. At this time, we think of the way in which two branch Lans of a group are connected in a real-life scenario. In fact, there is also a virtual device vEth like a network cable in Linux (does it feel that Linux is omnipotent at this time? ), known as Virtual Ethernet Device, is a virtual device similar to an Ethernet network

VEth has the following characteristics:

As a virtual Ethernet device, vEth can connect two different Network Namespace.

VEth is always created in pairs, so it is generally called veth pair. Because there is no network cable with only one end.

When one end of the vEth receives a packet, the other end receives it immediately.

The peer interface of vEth can be found through ethtool. (note that it will be used later)

Understanding the above points is much easier for us to understand the network communication between containers.

VEth practical operation

Create a vEth:

Ip link add

Type veth peer name

Create a vEth device named veth0A with the peer veth0B.

You can see that there are two more network interfaces veth0A and veth0B in the root network namespace, and the name of the network interface @ is followed by the interface name of the opposite end.

Create Network Namespace netA and netB:

Add interface veth0A to netA and interface veth0B to netB:

Ip link set veth0A netns netAip link set veth0B netns netB

At this time, when you look at the root network namespace network interface through IP a, you can find that you can no longer see the interfaces veth0A and veth0B (an interface can only be under one Network Namespace at a time).

If you look at the two Network Namespace of netA and netB respectively, you can see that there is one more network interface in both Network Namespace.

Pull up the two network interfaces and match them with IP. Here you will configure IP 192.168.100.1 and IP 192.168.100.2 for veth0A:

Ip netns exec netA ip link set veth0A upip netns exec netA ip addr add 192.168.100.1/24 dev veth0A

Ip netns exec netB ip addr add 192.168.100.2/24 dev veth0B

Test the network connection between the two Network Namespace netA and netB connected through veth pair.

In netA (192.168.100.1), ping netB (192.168.100.2):

In netB (192.168.100.2), ping netA (192.168.100.1):

It can be found that the two Network Namespace of netA and netB can communicate normally after connecting through veth pair.

After solving the problem of container Network Namespace isolation, students who have experience in cloud computing or are familiar with OpenStack and ZStack will find that the current scenario is exactly the same as the network interconnection between virtual machines.

As a layer 2 network device, what should vEth do when it needs to be connected to other network devices? In real-life scenarios, we use a switch to connect different network cables. In fact, this is also the case in the virtualization scenario. Linux Bridge and Open vSwith (OVS) are two commonly used solutions to connect layer 2 networks, and Linux Bridge is used in Docker.

| | Docker and Kubernetes |

As a container orchestration platform, Kubernetes can choose either Docker or rkt in terms of container engine. Here, containers are created directly through Docker and Kubernetes for simple comparison. When Kubernetes creates a Pod, it first creates a pause container. Its only function is to retain and occupy a network namespace (Network Namespace) shared by all containers in the Pod. In this way, a Pod IP does not change with the start and stop of a container in the Pod.

Container Network under Docker

Let's take a look at what it looks like without Kubernetes. When Docker starts, a bridge named docker0 will be created by default, and the network segment used by default is 172.17.0.0and16, and each container created on this node will be assigned an IP under this network segment. Containers communicate with each other by connecting to the docker0.

Create two containers manually:

Docker run-it-name testA busybox shdocker run-it-name testB busybox sh

Check the status of the network interface.

Container testA:

Container testB:

View the bridge status:

You can find that docker0 has two virtual network interfaces (vEth) connected to it.

Grab the package via tcpdump on docker0:

Tcpdump-n-I docker0

It can be found that container testA and container testB forward network packets through the docker0 bridge.

Container network after joining Kubernetes

In fact, after joining Kubernetes, the container network communication mode has not changed, but for the consideration of network address planning, Kubernetes has re-created a bridge cni0 to replace docker0 to be responsible for the allocation of network addresses on this node, while the actual network segment management is handled by Flannel.

Let's take the creation of two Pod running BusyBox images as examples.

First, add label to the two work node in the Kubernetes cluster to facilitate the scheduling of Pod to the same node for testing:

[root@10-10-88-192 network] # kubectl get node-- show-labelsNAME STATUS ROLES AGE VERSION LABELS10-10-88-170 Ready 47d v1.10.5-28+187e1312d40a02 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=10-10-88-17010-10-88-192 Ready master 47d v1.10.5-28+187e1312d40a02 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=10-10-88-192 Node-role.kubernetes.io/master=10-10-88-195 Ready 47d v1.10.5-28+187e1312d40a02 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux Kubernetes.io/hostname=10-10-88-195 [root@10-10-88-192 network] # [root@10-10-88-192 network] # kubectl label-- overwrite node 10-10-88-170 host=node1node "10-10-88-170" labeled [root@10-10-88-192 network] # [root@10-10-88-192 network] # kubectl label-- overwrite node 10-10-88-195 host=node2node "10-10-88-195" labeled [root@10-10-88- 192 network] # [root@10-10-88-192 network] # kubectl get node-- show-labelsNAME STATUS ROLES AGE VERSION LABELS10-10-88-170 Ready 47d v1.10.5-28+187e1312d40a02 beta.kubernetes.io/arch=amd64 Beta.kubernetes.io/os=linux,host=node1,kubernetes.io/hostname=10-10-88-17010-10-88-192 Ready master 47d v1.10.5-28+187e1312d40a02 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=10-10-88-192 Ready 47d v1.10.5-28+187e1312d40a02 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=10-10-88-192 network] #

Create two Pod and schedule them to the same node (host1) by adding nodeSelector.

Edit the yaml configuration file for Pod:

Create a Pod based on the yaml file:

You can see that both Pod are scheduled to the node 10-10-88-170 as expected.

From the IP a command, you can see that there are two more vethXXX-style network interfaces on the host of Pod:

[root@10-10-88-170170] # ip A1: lo: mtu 65536 qdisc noqueue state UNKNOWN qlen 1link/loopback 0000 qdisc noqueue state UNKNOWN qlen 1link/loopback 0000 brd 00:00:00:00:00:00inet 127.0.0.1 8 scope host lo valid_lft forever preferred_lft foreverinet6:: 1 mtu 128 scope host valid_lft forever preferred_lft forever2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000link/ether fa:35:b6:5e:ac:00 brd ff:ff : ff:ff:ff:ffinet 10.10.88.170/24 brd 10.10.88.255 scope global eth0 valid_lft forever preferred_lft foreverinet6 fe80::f835:b6ff:fe5e:ac00/64 scope link valid_lft forever preferred_lft forever3: eth2: mtu 1500 qdisc pfifo_fast state UP qlen 1000link/ether fa:88:2a:44:2b:01 brd ff:ff:ff:ff:ff:ffinet 172.16.130.164/24 brd 172.16.130.255 scope global eth2 Valid_lft forever preferred_lft foreverinet6 fe80::f888:2aff:fe44:2b01/64 scope link valid_lft forever preferred_lft forever4: docker0: mtu 1500 qdisc noqueue state DOWNlink/ether 02:42:43:a1:fc:ad brd ff:ff:ff:ff:ff:ffinet 172.17.0.1/16 scope global docker0 valid_lft forever preferred_lft forever5: flannel.1: mtu 1450 qdisc noqueue state UNKNOWNlink/ether 0e:c4:2c:84:a5:ea brd ff:ff:ff:ff : ff:ffinet 10.244.2.0/32 scope global flannel.1 valid_lft forever preferred_lft foreverinet6 fe80::cc4:2cff:fe84:a5ea/64 scope link valid_lft forever preferred_lft forever6: cni0: mtu 1450 qdisc noqueue state UP qlen 1000link/ether 0a:58:0a:f4:02:01 brd ff:ff:ff:ff:ff:ffinet 10.244.2.1/24 scope global cni0 valid_lft forever preferred_lft foreverinet6 fe80::f0a0:7dff:feec:3ffd/64 Scope link valid_lft forever preferred_lft forever9: veth3a69de99@if3: mtu 1450 qdisc noqueue master cni0 state UPlink/ether 86:70:76:4f:de:2b brd ff:ff:ff:ff:ff:ff link-netnsid 2inet6 fe80::8470:76ff:fe4f:de2b/64 scope link valid_lft forever preferred_lft forever10: vethc8ca82e9@if3: mtu 1450 qdisc noqueue master cni0 state UPlink/ether 76:ad:89:ae:21:68 brd ff:ff:ff:ff:ff:ff link-netnsid 3inet6 Fe80::74ad:89ff:feae:2168/64 scope link valid_lft forever preferred_lft forever39: veth786e1634@if3: mtu 1450 qdisc noqueue master cni0 state UPlink/ether 66:99:fe:30:d2:e1 brd ff:ff:ff:ff:ff:ff link-netnsid 4inet6 fe80::6499:feff:fe30:d2e1/64 scope link valid_lft forever preferred_lft forever40: vethef16d6b0@if3: mtu 1450 qdisc noqueue master cni0 state UPlink/ether c2:7f:73:93:85:fc brd ff:ff Ff:ff:ff:ff link-netnsid 5inet6 fe80::c07f:73ff:fe93:85fc/64 scope link valid_lft forever preferred_lft forever [root@10-10-88-170] # [root@10-10-88-170] # brctl showbridge name bridge id STP enabled interfacescni0 8000.0a580af40201 no veth3a69de99 veth786e1634 vethc8ca82e9 vethef16d6b0docker0 8000.024243a1fcad no [root@10-10-88-170] #

The network connection between the two Pod is shown in the figure:

The process for sending network packets from Container A to Container B is as follows:

1. The network packet is sent from the eth0 of the busybox1 and enters the root netns through the vethef16d6b0 (the network packet is sent from one end of the vEth and immediately received by the other end).

two。 The network packet is passed to the bridge cni0, which sends "who has this IP?" To discover the destination to which the network packet needs to be forwarded (10.244.2.208).

3. Busybox2 replied that it had this IP, so the bridge knew that the network packet should be forwarded to veth786e1634 (busybox2).

4. The network packet arrives at the veth786e1634 interface and enters the netns of the busybox2 through vEth, thus completing the process of the network packet from one container busybox1 to another container busybox2.

Students who have doubts about the above process can also verify the conclusion on their own. The best way is to grab packets on each network interface through the tcpdump command to see how the network packets pass through the bridge and then flow from veth pair to another container network.

Container network depends to a large extent on the development of virtual network, which is the trend of technological development, which is called standing on the shoulders of giants.

On how to analyze the container network in Kubernetes to share here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.