Shulou(Shulou.com)06/02 Report--

PouchContainer is Alibaba Group's open source efficient, lightweight enterprise-level rich container engine technology, with strong isolation, high portability, low resource consumption and other features. It can help enterprises to quickly realize the containment of inventory business, and at the same time improve the utilization of physical resources in super-large-scale data centers.

PouchContainer originates from Alibaba's internal scene. In the early days of its birth, Alibaba engineers devoted themselves to the design of how to protect Internet applications. The technical features of PouchContainer, such as strong isolation and rich containers, are the best proof. Under the mass scale of Alibaba, the support of PouchContainer for business has been tested by double 11. After open source, Ali Container has become an inclusive technology, which is positioned to "help enterprises quickly realize the containment of inventory business".

This article will introduce how PouchContainer implements network and how to connect containers to network. In order to fully illustrate the connection mechanism of network, this paper will take the Connect method as an example to describe how to dynamically connect an container to an existing network.

1. The mechanism of implementing network by PouchContainer

In the current container network virtualization technology, the CNM (Container Network Model) model implemented by Docker is a general solution. CNM constructs a mature container virtualization network model and defines a variety of standardized interfaces for developers to call. PouchContainer follows the CNM model and realizes the communication between containers based on libnetwork. Let's start with an introduction to the core components of the three CNM, Sandbox, Endpoint, and Network.

Sandbox

The word Sandbox is given different definitions in different mechanisms. For example, in CRI (container runtime interface), sandbox represents the concept of pod. In the CNM model, sandbox represents the network stack configuration of a container, including the network card, routing table, and DNS settings that manage the container. Sandbox can be implemented through the Linux system's network namespace, a FreeBSD Jail, or other similar concepts. A sandbox can contain multiple endpoints.

Endpoint

An endpoint connects the sandbox to the network. An endpoint can be implemented through veth pair,Open vSwitch internal port or other ways. The more common method is to use veth pair, as the name implies, veth pair must appear in pairs, so there will be two network cards: veth0 and veth2. When creating a container, one of the blocks will be set to the inside of the container and act as the eth0 inside the container. All packets with the destination address of container IP will pass through the eth0 ENI; the other piece (hereinafter referred to as veth device) will be connected to the bridge of the host. Packets from the veth device are forwarded to the corresponding eth0 device, and when the destination address of the packet is the IP of the eth0 device, it can be processed by the kernel protocol stack. Veth pair is used to connect two network namespace to establish a network connectivity relationship. An Endpoint can only belong to one Network and only one Sandbox.

Network

A Network is a collection of Endpoints that can communicate with each other. An network can be implemented through Linux bridge,VLAN or other means. It is worth mentioning that a network can contain many endpoints.

As you can see, under the structure shown in the following figure, Container An and Container B both belong to backend network, and these two container form a network connection through their respective purple endpoint; container B and container C both belong to frontend network, and a blue endpoint forms a network connection. Therefore, container An and container B can communicate, and container B and container C can also communicate.

Next, let's focus on the two endpoints inside container B. although both backend network and frontend network have their own corresponding endpoint within container B, there is no communication between the purple endpoint and the blue endpoint. So backend network and frontend network are two completely isolated network that do not connect to the same container. It is obvious that there is no communication between container An and container C.

2. PouchContainer built-in network mode 2.1 bridge mode

Bridge mode is the default network mode of PouchContainer. If you do not specify network mode, that is, if you do not write the-- net parameter, the container will be created in bridge mode. When pouchd starts, a virtual bridge p0 is automatically created on the host. When creating containers in bridge mode, pouchd selects an unused IP from the IP network segment where the p0 bridge is located and assigns it to the container's eth0 network card. The IP of p0 is the default gateway for these containers.

2.2 host mode

When starting the container, select host mode, then the container will not get a separate network namespace, but will share the network namespace with the host. Therefore, the container does not have its own network card and IP configuration, and will use the host's IP and port, but fs and pid are isolated from the host.

2.3 container mode

Containers created in container mode share a network namespace with existing containers, directly following their veth device pairs.

2.4 none mode

Containers created using none mode have a separate network namespace, but do not make any network configuration for the container. Therefore, it can be considered that containers in none mode do not communicate with other containers. However, after the container is created, you can add a network card to it and configure IP so that you can communicate with the container under the same network.

2.5 Conceptual intersection of CNM and network patterns

A network is a unique, recognizable endpoint group, and the endpoint within the group can communicate with each other. Compared with CNM, endpoint can be simply understood as a pair of veth devices. There can be multiple endpoints in the sandbox of a container, and each endpoint represents a connection to a specific network.

3. Process analysis of network connect

/ / daemon/mgr/container.go

/ / Connect is used to connect a container to a network.

Func (mgr * ContainerManager) Connect (ctx context.Context, name string, networkIDOrName string, epConfig * types.EndpointSettings) error {

……

If err: = mgr.updateNetworkConfig (c, n.Name, epConfig); err! = nil {

Return err

} else

If err: = mgr.connectToNetwork (ctx, c, networkIDOrName, epConfig); err! = nil {

Return err

}

Return c.Write (mgr.Store)

}

You can see that in the Connect function, you first get the specific container and network based on the parameters passed in. Among the epConfig parameters, the parameters passed through flag on the CLI are stored, such as the alias of container in a specific network, the specified IP range, and so on.

Check c.State.Status to determine the status of container at this time. Container of dead status cannot perform connect operation. For container that is not running but still live, simply call updateNetworkConfig () to update the network configuration of container, adding the incoming epConfig to the container's network configuration. In this case, no network card is assigned to the container, so the container is not successfully connected to the network. For the container with running status, call connectToNetwork () for subsequent operations. ConnectToNetwork () will configure the network card according to the given network and container, then assign a network card on the host, and finally add the network card to the sandbox of container. In this way, container is successfully connected to network! The specific process will be parsed later.

The function of c.Write (mgr.Store) is to connect container to a series of configurations on network and write them into the metadata of container, thus ensuring the persistence of data. Otherwise, the established network connection is only one-time, and all data and related configurations will be lost after pouchd restart.

/ / daemon/mgr/container.go

Func (mgr * ContainerManager) connectToNetwork (ctx context.Context, container * Container, networkIDOrName string, epConfig * types.EndpointSettings) (err error) {

……

Endpoint: = mgr.buildContainerEndpoint (container)

……

If _, err: = mgr.NetworkMgr.EndpointCreate (ctx, endpoint); err! = nil {

……

}

Return mgr.updateNetworkConfig (container, networkIDOrName, endpoint.EndpointConfig)

}

Endpoint contains three parts of information, part of the information comes from container, part of the information comes from network, and the last part of information is the configuration of flag in the connect command. The logic of buildContainerEndpoint () is relatively simple, which is to get the container-related information needed by endpoint. The EndpointCreate () of NetworkMgr is then called to do the concrete build.

/ / daemon/mgr/network.go

/ / EndpointCreate is used to create network endpoint.

Func (nm * NetworkManager) EndpointCreate (ctx context.Context, endpoint * types.Endpoint) (string, error) {

……

/ / create endpoint

EpOptions, err: = endpointOptions (n, endpoint)

……

EndpointName: = containerID [: 8]

Ep, err: = n.CreateEndpoint (endpointName, epOptions...)

……

/ / create sandbox

Sb: = nm.getNetworkSandbox (containerID)

If sb = = nil {

SandboxOptions, err: = buildSandboxOptions (nm.config, endpoint)

……

Sb, err = nm.controller.NewSandbox (containerID, sandboxOptions...)

……

}

/ / endpoint joins into sandbox

JoinOptions, err: = joinOptions (endpoint)

……

If err: = ep.Join (sb, joinOptions...); err! = nil {

Return "", fmt.Errorf ("failed to join sandbox (% v)", err)

}

/ / update endpoint settings

EpInfo: = ep.Info ()

If epInfo.Gateway ()! = nil {

EndpointConfig.Gateway = epInfo.Gateway () .String ()

}

If epInfo.GatewayIPv6 () .To16 ()! = nil {

EndpointConfig.IPV6Gateway = epInfo.GatewayIPv6 () .String ()

}

Endpoint.ID = ep.ID ()

EndpointConfig.EndpointID = ep.ID ()

EndpointConfig.NetworkID = n.ID ()

Iface: = epInfo.Iface ()

……

Return endpointName, nil

}

The whole process of creating endpoint is implemented by calling libnetwork. First, call endpointOptions () to build the EndpointOption parameter required by the interface, and this parameter of the setter function type can pass different option to the interfaces of network and endpoint. Then call the libnetwork's

The CreateEndpoint () interface is used for specific construction. The actual work performed by CreateEndpoint () includes assigning the IP and interface (Iface) to the endpoint, and the corresponding configuration is applied to the Endpoint, including the configuration rules and port information of the iptables.

Sandbox represents network namespace, which is unique to container, and its creation is also based on libnetwork. Sandbox contains iconic information for container to establish network communication, such as IP address, Mac address, routing and DNS configuration. The existing sandbox is traversed to determine whether the corresponding sandbox exists, and if so, the corresponding sandbox is returned directly. In none mode, container follows the host's namespace, and the returned sandbox is empty, and a new sandbox is created. The process of creating a sandbox is to call namespace and cgroup to create a separate sandbox space.

The operation of adding endpoint to sandbox is actually the process of assigning the network card to container and injecting the network resources allocated by endpoint into sandbox. The network card is the core of establishing the connection, and the container connects to the network through the virtual network card, thus communicating with other container.

The final step is to synchronously update the changes to the endpoint configuration.

4. Summary

Reviewing the entire process of establishing a network connection can be simply broken down into several steps. Container needs a unique network namespace to mark itself when communicating, so there must be the creation of sandbox; the realization of communication needs a network card as the basis, then there must be the creation of endpoint; finally, endpoint is added to the sandbox to establish the basis for communication between containers, and the connection is successfully completed.

To learn more about PouchContainer, please visit https://pouchcontainer.io

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.