Shulou(Shulou.com)06/02 Report--

Different from the default network model of Docker, Kubernetes has formed its own network model, which is more suitable for the traditional network model, and the application can migrate smoothly from the non-container environment to the Kubernetes environment.

Since the emergence of the Docker container, the network communication of the container has been the focus of attention, and the network solution of the container can be divided into two parts:

Communication between containers of a single host; communication between containers across hosts. 1. Single host Docker network communication

Using Net Namespace, you can create an isolated network environment for the Docker container, which has a completely independent network stack and is isolated from the host. You can also make Docker containers share the network namespaces of hosts or other containers.

When we use docker run to create a Docker container, we can use the-- network= option to specify the network mode of the container. Docker has the following four network modes:

Host mode, specified using-- network=host, does not support multi-hosts; bridge mode, specified using-- network=bridge, default setting, does not support multi-hosts; container mode, specified using-- network=container:NAME_or_ID, that is, joiner containers, does not support multi-hosts; none mode, specified using-- network=none, does not support multi-hosts. 1.1.The host mode

Containers connected to the host network share the network stack of Docker host, and the network configuration of the container is exactly the same as host.

Let's take a look at the network of the host.

[root@datanode03 ~] # ifconfigdocker0: flags=4099 mtu 1500 inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255 ether 02:42:44:8d:48:70 txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0enp1s0: flags=4163 mtu 1500 inet 192.168.1.203 netmask 255.255.255.0 broadcast 192.168.1.255 inet6 fe80::2e0:70ff:fe92:4779 prefixlen 64 scopeid 0x20 ether 00:e0:70:92:47:79 txqueuelen 1000 (Ethernet) RX packets 46093 bytes 66816291 (63.7 MiB) RX errors 0 dropped 1 overruns 0 frame 0 TX packets 24071 bytes 1814769 (1.7 MiB) TX errors 0 dropped 0 overruns 0 Carrier 0 collisions 0lo: flags=73 mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6:: 1 prefixlen 128 scopeid 0x10 loop txqueuelen 0 (Local Loopback) RX packets 170 bytes 107720 (105.1 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 170 bytes 107720 (105.1 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Then create a container for the host network and view the network information of the container.

[root@datanode03 ~] # docker run-it-- network=host busyboxUnable to find image 'busybox:latest' locallylatest: Pulling from library/busybox90e01955edcd: Pull complete Digest: sha256:2a03a6059f21e150ae84b0973863609494aad70f0a80eaeb64bddd8d92465812Status: Downloaded newer image for busybox:latest/ # ifconfigdocker0 Link encap:Ethernet HWaddr 02:42:44:8D:48:70 inet addr:172.17.0.1 Bcast:172.17.255.255 Mask:255.255.0.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets: 0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0B) TX bytes:0 (0.0B) enp1s0 Link encap:Ethernet HWaddr 00:E0:70:92:47:79 inet addr:192.168.1.203 Bcast:192.168.1.255 Mask:255.255.255.0 Inet6 addr: fe80::2e0:70ff:fe92:4779/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:45850 errors:0 dropped:1 overruns:0 frame:0 TX packets:23921 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:66794758 (63.7 MiB) TX bytes:1783655 (1.7 MiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: 1Comp128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:170 errors:0 dropped:0 overruns:0 frame:0 TX packets:170 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:107720 (105.1 KiB) TX bytes:107720 (105.1 KiB)

You can see all the network cards of host in the container, and even the hostname is host. You can directly use the host IP address to communicate with the outside world without additional NAT translation. Because when the container communicates, it no longer needs to be forwarded or encapsulated by Linux Bridge and other methods, so it has a great advantage in performance.

Of course, the Host model has both advantages and disadvantages, including the following disadvantages:

The container does not have an isolated and independent network stack: the container competes for network resources because it shares the network stack with the host, and the collapse of the container may also cause the host to crash, which is not allowed in the production environment. Port resources: ports that are already in use on Docker host can no longer be used. 1.2 Bridge mode

Bridge mode is the default network mode of Docker, and it is also the most commonly used network mode for developers. In this mode, Docker creates an independent network stack for the container, ensures that the operations within the container use an independent network environment, and realizes the network stack isolation between containers and between containers and hosts. At the same time, through the Docker0 bridge on the host, the container can communicate with the host and even the outside world.

As can be seen from the figure above, containers can communicate with hosts and other external machines. On the same host, containers are all bridged on the Docker0 bridge, and Docker0 acts as a virtual switch to communicate with containers. However, because the IP address of the host is not on the same network segment as the IP address of the container veth pair, the technology of veth pair and NameSpace alone is not enough to make the network outside the host actively discover the existence of the container. Docker uses port binding (through the NAT of iptables) to forward port traffic on the host to the port in the container, so that the outside world can communicate with the processes in the container.

1.3 Container mode

Container mode is a special network mode. Containers in this mode use the network namespaces of other containers, and network isolation is between Bridge mode and Host mode. That is, when a container shares a network namespace with other containers, there is no network isolation between the two containers, but they are also isolated from other containers of the host machine.

Containers in Container mode can communicate with other containers in the same network namespace through localhost, and the transmission efficiency is high. This mode saves a certain amount of network resources, but does not change the way the container communicates with the outside world. The concept of Pod is introduced under the Kubernetes architecture. Kubernetes creates an infrastructure container for Pod. Other containers under the same Pod share the network namespace of this infrastructure container in Container mode and access each other as localhost to form a unified whole.

1.4.The None mode

Unlike the previous ones, the Docker container in None mode has its own Network Namespace, but there is no network configuration for the Docker container. In other words, the Docker container does not have network card, IP, routing and other information. Users are required to add network cards, configure IP, etc., for the Docker container.

Second, cross-host Docker network communication classification 2.1communication scheme

The common cross-host communication schemes are as follows:

Host mode: the container uses the host network directly so that it can naturally support cross-host communication. Although this method can solve the problem of cross-host communication, the application scenario is very limited, port conflicts are easy to occur, and it is impossible to isolate the network environment. The collapse of one container is likely to cause the collapse of the entire host. Port mode: access the services in the container by binding the container port to the host port and using the host IP + port when communicating across hosts. Obviously, this method can only support applications with layer 4 or above of the network stack, and the container is tightly coupled with the host, so it is difficult to deal with the problem flexibly and the scalability is not good; define the container network: use third-party SDN tools such as Open vSwitch or Flannel to build a network environment for the container to communicate across hosts. This kind of scheme generally requires that the cidr of the Docker0 bridge on each host is different, in order to avoid the problem of IP conflict and limit the range of IP available to the container on the host. And when the container needs to provide services outside the cluster, it needs a more complex configuration, which requires high network skills of the deployment implementation personnel. 2.2. Container network specification

With the development of container network, two camps have been formed:

CNI dominated by CNM;Google, CoreOS and Kubernetes of Docker.

CNM and CNI are network specifications or network architecture, not network implementation, so they do not care about the network implementation of the container. CNM and CNI are only concerned with network management.

CNM (Container Network Model): the advantage of CNM is that the native, container network and Docker container lifecycle are closely integrated; the disadvantage is that it is "kidnapped" by Docker. Container network implementations that support CNM network specification include: Docker Swarm overlay, Macvlan & IP networkdrivers, Calico, Contiv, Weave and so on. CNI (Container Network Interface): the advantage of CNI is that it is compatible with other container technologies (rkt) and the upper orchestration system (Kubernetes&Mesos), and the community is very active; the disadvantage is that it is not native to Docker. Container network implementations that support CNI network specifications include: Kubernetes, Weave, Macvlan, Calico, Flannel, Contiv, Mesos CNI, because they all implement the CNI specification. No matter which scheme the user chooses, the network model is the same, that is, each Pod has an independent IP that can communicate directly. The difference lies in the underlying implementation of different schemes, some using VxLAN-based Overlay implementation, some are Underlay, there are differences in performance. Then there is whether or not to support Network Policy. 2.3. Implementation scheme of network communication

However, from the point of view of network implementation, it can be divided into:

Tunnel scheme: tunnel scheme is also widely used in IaaS layer network, its main disadvantage is that the complexity will increase with the increase of node size, and it is more difficult to track after network problems, which is a problem to be considered in the case of large-scale clusters.

Weave:UDP broadcast, the local establishment of a new BR, through PCAP interworking. Open vSwitch (OVS): based on VxLAN and GRE protocols, but the performance loss is serious. Flannel:UDP Radio, VxLAN. Racher:IPSec .

Routing scheme: generally based on layer 3 or layer 2 to achieve network isolation and interworking across host containers, problems are also easy to troubleshoot.

Calico: a routing scheme based on BGP protocol, which supports fine ACL control (Nerwork Policy) and has high affinity for hybrid clouds.

Macvlan: from the perspective of logic and Kernel layer, it is the best solution of isolation and performance. Based on layer 2 isolation, it needs layer 2 router support, which is not supported by most cloud service providers, so it is difficult to implement on hybrid cloud.

2.4.The Kubernetes network model

Kubernetes adopts a network model based on flat address space. Each Pod in the cluster has its own IP address, and the Pod can communicate directly without configuring NAT. In addition, containers in the same Pod share the IP of Pod and can communicate over localhost.

This network model is very friendly to application developers and administrators, and applications can be easily migrated from traditional networks to Kubernetes. Each Pod can be seen as an independent system, while containers in Pod can be seen as different processes in the same system.

Communication between containers in Pod

When Pod is dispatched to a node, all containers in Pod run on that node, and these containers share the same local file system, IPC, and network namespace.

There is no port conflict between different Pod because each Pod has its own IP address. When a container uses localhost, it means that the address space of the Pod to which the container belongs is used.

For example, Pod A has two containers, container-A1 and container-A2,container-A1, listening on port 1234. When container-A2 connects to localhost:1234, it is actually accessing container-A1. This does not conflict with Pod B on the same node, even though the container container-B1 in Pod B is listening on port 1234.

Communication between Pod

The IP of Pod is cluster visible, that is, any other Pod and nodes in the cluster can communicate with Pod directly through IP, which does not require any network address translation, tunneling or proxy technology. Pod uses the same IP internally and externally, which also means that standard naming services and discovery mechanisms, such as DNS, can be used directly.

Communication between Pod and Service

Pod can communicate directly through the IP address, but only if the Pod knows the other party's IP. In Kubernetes clusters, Pod may be destroyed and created frequently, which means that the IP of Pod is not fixed. To solve this problem, Service provides an abstraction layer to access Pod. No matter how the back-end Pod changes, Service provides services as a stable front-end. At the same time, Service also provides high availability and load balancing, and Service is responsible for forwarding requests to the correct Pod.

External access

Whether it is Pod's IP or Service's Cluster IP, they are only visible in the Kubernetes cluster, and these IP are private to the world outside the cluster.

Third, cross-host Docker network 3.1 Flannel network scheme

Kubernetes installation mode.

Kubectl apply-f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml

Flannel is a container network solution developed by CoreOS. Flannel assigns a subnet to each host, and the container assigns IP from this subnet. These IP can be routed between host, and the containers can communicate across hosts without NAT and port mapping.

Each subnet is divided from a larger IP pool, and flannel runs an agent called flanneld on each host, which is responsible for allocating subnet from the pool. In order to share information among hosts, flannel uses etcd (key-value distributed database similar to consul) to store network configuration, allocated subnet, IP of host, and so on.

How packets are forwarded between hosts is implemented by backend. Flannel provides a variety of backend, including UDP, vxlan, host-gw, aws-vpc, gce, and alloc routing, the most commonly used being vxlan and host-gw.

Flannel is essentially an overlay network (Overlay Network), that is, TCP data is packaged in another network packet for routing, forwarding and communication.

The container directly uses the IP access of the target container, which is sent by default through the eth0 inside the container; the message is sent to the vethXXX;vethXXX through veth pair, which is directly connected to the cni0, and the message is sent through the virtual bridge cni0; look up the routing table, and the messages from the external container IP are forwarded to the virtual network card of flannel.1, which is a P2P virtual network card, and then the message is forwarded to the flanneld at the other end of the monitor. Flanneld maintains the routing table between each node through etcd, encapsulates the original message UDP and sends it out through the configured iface; the message finds the target host through the network stack between hosts; the message continues to be sent up, reaches the transport layer, and is processed by the listening flanneld program; the data is unpackaged and then sent to the flannel.1 virtual network card; the routing table is searched and it is found that the message of the corresponding container should be handed over to cni0 Cni0 connects to its own container and sends the message.

The default backend for the flannel we installed using kubectl apply is vxlan,host-gw, which is another backend for flannel, and we switch the previous vxlan backend to host-gw.

Unlike vxlan, host-gw does not encapsulate packets, but instead creates route entries to other host subnet in the host's routing table, allowing containers to communicate across hosts. To use host-gw, first modify the configuration flannel-config.json of flannel:

Kubectl edit cm kube-flannel-cfg-o yaml-n kube-system

Find the following fields to modify.

Net-conf.json: | {"Network": "10.244.0.0 host-gw 16", "Backend": {"Type": "host-gw"}}

Host-gw configures each host as a gateway, and the host knows the subnet and forwarding addresses of other hosts. Because vxlan requires additional packaging and unpacking of packets, the performance is better than vxlan.

3.2.The Calico network scheme

Kubernetes installation mode.

Kubectl apply-f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yamlkubectl apply-f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

Calico regards the protocol stack of each operating system as a router, and then thinks that all containers are network terminals connected to this router, runs the standard routing protocol BGP between routers, and then lets them learn how to forward this network topology, so Calico is a pure three-layer virtual network scheme. Calico assigns an IP to each container, and each host is a router. Connect containers of different host. Unlike VxLAN, Calico does not do additional encapsulation of packets, does not require NAT and port mapping, and has good scalability and performance.

Compared with other container network solutions, Calico also has one major advantage: network policy. Users can dynamically define ACL rules and control the data packets entering and leaving the container to achieve business requirements.

Felix: agent process running on each Host, mainly responsible for network interface management and monitoring, routing, ARP management, ACL management and synchronization, status reporting, etc. Orchestrator Plugin: orchestration plug-ins are not some processes that run independently, but design plug-ins integrated with platforms such as K8s and OpenStack. For example, Neutron's ML2 plugin is used for users to use Neutron API to manage Calico, the essence is to solve the compatibility between model and API. Etcd:Calico model storage engine; BGP Client (BIRD): Calico deploys a BGP Client for each Host and implements it using BIRD. BIRD is a separate continuous development project that implements many dynamic routing protocols such as BGP, OSPF, RIP and so on. The role of Calico is to listen to the routing information injected by Felix on Host, and then broadcast to the remaining Host nodes through BGP protocol, so as to realize network interworking. BGP Route Reflector (BIRD): in large-scale networks, the scheme of using BGP client alone to form mesh full-network interconnection will lead to scale restrictions, because two nodes are interconnected and N ^ 2 connections are needed. In order to solve this scale problem, BGP's Router Reflector method can be used to make all BGP Client interconnect and synchronize routes with specific RR nodes only, thus greatly reducing the number of connections.

3.3.The Canal network scheme

Network Policy

Network Policy is a resource of Kubernetes. Network Policy selects Pod through Label and specifies how other Pod or outsiders communicate with these Pod.

By default, all Pod is unquarantined, that is, network traffic from any source can access the Pod without any restrictions. When Network Policy is defined for Pod, only traffic allowed by Policy can access Pod.

However, not all Kubernetes network schemes support Network Policy. For example, Flannel does not support it, but Calico does. Next we will use Canal to demonstrate Network Policy. Canal is an interesting open source project that uses Flannel to implement Kubernetes cluster networks and Calico to implement Network Policy.

Kubectl apply-f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/canal/rbac.yamlkubectl apply-f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/canal/canal.yaml3.4, Docker overlay network scheme

Please check my blog https://blog.51cto.com/wzlinux/2112061.

Docker macvlan network scheme docker network create-d macvlan-- subnet=172.16.86.0/24-- gateway=172.16.86.1-o parent=enp0s9 mac_net1

Macvlan itself is a linxu kernel module, and its function is to allow multiple MAC addresses, that is, multiple interface, to be configured on the same physical network card, and each interface can configure its own IP. Macvlan is essentially a network card virtualization technology, so it's not surprising that Docker uses macvlan to implement container networks.

The biggest advantage of macvlan is its excellent performance. Compared to other implementations, macvlan does not need to create a Linux bridge, but connects directly to the physical network through an Ethernet interface.

Macvlan monopolizes the network card of the host, which means that a network card can only create one macvlan network:

But the number of network cards of the host is limited, how to support more macvlan networks? Docker network create-d macvlan-o parent=enp0s9 mac_net2

Fortunately, macvlan can connect not only to interface (such as enp0s9), but also to sub-interface (such as enp0s9.xxx).

VLAN is a commonly used network virtualization technology in modern networks. It can divide the physical layer 2 network into as many as 4094 logical networks. These logical networks are isolated at layer 2. Each logical network (i.e. VLAN) is distinguished by VLAN ID, and the value of VLAN ID is 1-4094.

The network card of Linux can also support VLAN (apt-get install vlan). The same interface can send and receive packets of multiple VLAN, but only if the sub-interface of VLAN is created.

For example, if you want enp0s9 to support both VLAN10 and VLAN20, you need to create sub-interface enp0s9.10 and enp0s9.20.

On the switch, if a port can only send and receive data from a single VLAN, the port is in Access mode, and if multiple VLAN is supported, it is Trunk mode, so the premise of the next experiment is:

The enp0s9 should be connected to the trunk port of the switch. However, we use a VirtualBox virtual machine, so there is no need for additional configuration.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.