Shulou(Shulou.com)06/02 Report--

PouchContainer is Alibaba Group's open source efficient, lightweight enterprise-level rich container engine technology, with strong isolation, high portability, low resource consumption and other features. It can help enterprises to quickly realize the containment of inventory business, and at the same time improve the utilization of physical resources in super-large-scale data centers. It has helped Alibaba Group to realize 100% containerization of online business, and the scale of double 11 containers has reached one million.

Background

Within Alibaba Group, a large part of the use of containers is the rich container mode, such as this rich container based on the traditional virtual machine operation and maintenance mode, in which a certain number of containers are still stateful. The update and upgrade of stateful service is a daily operation with high frequency within the enterprise. For the container technology delivered by image, the corresponding container operation is actually two steps: the deletion of old image container and the creation of new image container. The upgrade of stateful service requires that the new container must inherit all the resources of the old container, such as network, storage and other information. The following are two actual business cases to visually illustrate the requirements of rich container business publishing scenarios:

Customer case 1: when a database service is created for the first time, the remote data will be downloaded locally as the initial data of the database. Because the database initialization process is relatively long, in the possible service upgrade process, the new container needs to inherit the stored data from the old container to reduce the business release time.

Customer case 2: for a middleware service, the business adopts the service registration mode, that is, all the newly expanded container IP must first be registered in the server list, otherwise the new expanded business container will not be available. When the business container is upgraded and published each time, you need to ensure that the new container inherits the old container IP, otherwise the newly published service will become unavailable.

Nowadays, many enterprises use Moby (Docker renamed Moby in 2017) as the container engine, but there is not an interface in all API of Moby to upgrade standard containers. The way of combining API will inevitably increase the number of API requests, such as adding or deleting API of request container, API that needs to be retained by IP, etc., and may also increase the risk of failed upgrade operation.

Based on the above background, PouchContainer provides a upgrade interface at the container engine level to implement the container upgrade function. Sinking the container upgrade function to the container engine layer makes it more convenient to manipulate container-related resources, reduces a lot of API requests, and makes the container upgrade operation more efficient.

The concrete realization of Upgrade function

Introduction to the underlying storage of containers

The underlying interface of PouchContainer is Containerd v1.0.3. Compared with Moby, there is a big difference in container storage architecture. Before introducing how to upgrade containers in place in PouchContainer, it is necessary to briefly introduce the storage architecture of a container in PouchContainer:

Image.png | center | 600x336.3525091799266

Compared with the container storage architecture in Moby, PouchContainer is mainly different:

Without the concept of GraphDriver and Layer in PouchContainer, Snapshotter and Snapshot are introduced into the new storage architecture, thus more embracing the architectural design of containerd, a CNCF project. Snapshotter can be understood as a storage driver, such as overlay, devicemapper, btrfs, etc. Snapshot is a mirror snapshot, which is divided into two types: one is read-only, that is, the read-only data of each layer of the container image; the other is read-write, that is, the container read-write layer, in which all container incremental data is stored in read-write Snapshot

The container and image metadata in Containerd are stored in boltdb. The advantage is that each service restart does not need to initialize the container and image data by reading the host file directory information, but only needs to initialize boltdb.

Upgrade functional requirements

At the beginning of each system and function design, it is necessary to investigate in detail what pain points the system or function needs to solve for the user. After investigating the specific business scenarios in which Ali uses the container upgrade feature in place, we summarize three requirements for upgrade function design:

Data consistency

Flexibility

Robustness

Data consistency means that some data need to remain unchanged before and after upgrade:

Network: the container network configuration should remain unchanged before and after the upgrade

Storage: the new container needs to inherit all the volume of the old container

Config: the new container needs to inherit some configuration information from the old container, such as Env, Labels, etc.

Flexibility means that upgrade operations allow the introduction of new configurations based on the old containers:

Allows you to modify the cpu, memory and other information of the new container

For new images, you should not only support the specification of a new Entrypoint, but also allow inheritance of the Entrypoint of the old container

You can add a new volume to the container. The new image may contain new volume information. When you create a new container, you need to parse this volume information and create a new volume.

Robustness means that during the container upgrade operation, possible exceptions need to be handled, and the rollback strategy is supported. The upgrade failure can be rolled back to the old container.

The concrete realization of Upgrade function

Upgrade API definition

First, let's explain the definition of the upgrade API entry layer, which is used to define which parameters of the container can be modified by the upgrade operation. As defined by ContainerUpgradeConfig, the container upgrade operation can operate on both ContainerConfig and HostConfig of the container. If you refer to the definitions of these two parameters in the apis/types directory of the PouchContainer github code repository, you can find that the upgrade operation can actually modify all relevant configurations of the old container.

/ / ContainerUpgradeConfig ContainerUpgradeConfig is used for API "POST / containers/upgrade".

/ / It wraps all kinds of config used in container upgrade.

/ / It can be used to encode client params in client and unmarshal request body in daemon side.

/ /

/ / swagger:model ContainerUpgradeConfig

Type ContainerUpgradeConfig struct {

ContainerConfig

/ / host configHostConfig * HostConfig `json: "HostConfig,omitempty" `

}

Detailed operation flow of Upgrade

Container upgrade operation is actually the process of deleting an old container and creating a new container using a new image while ensuring that the network configuration and the original volume configuration remain unchanged. The operation flow of upgrade is described in detail as follows:

First of all, you need to back up all the operations of the original container, which is used to roll back after the upgrade fails.

Update the container configuration parameters and merge the new configuration parameters in the request parameters into the old container parameters to make the new configuration effective

Special handling of the image Entrypoint parameter: if the Entrypoint parameter is specified in the new parameter, the new parameter is used; otherwise, check the Entrypoint of the old container, and if this parameter is specified by the configuration parameter rather than provided in the old image, the Entrypoint of the old container is used as the Entrypoint of the new container; if neither, use the Entrypoint in the new image to create the new container's Entrypoint. The reason for this treatment of the new container Entrypoint is to maintain the continuity of the container service entry parameters.

Judge the state of the container. If it is a container in running state, first stop the container; then create a new Snapshot based on the new image as the read-write layer of the new container

After the new Snapshot is created successfully, judge the status of the old container before upgrade again. If the status is running, you need to start the new container, otherwise no action is required.

Finally, clean up the container upgrade, delete the old Snapshot, and save the latest configuration.

Upgrade operation rollback

Some exceptions may occur in the upgrade operation. The current upgrade strategy is to roll back to the original state of the old container when an exception occurs. Here, we need to first define the upgrade failure:

Failed to create a new resource for the new container. You need to perform a rollback operation: when creating a new Snapshot,Volumes and other resources for the new container, the rollback operation will be performed.

When a system error occurs when starting a new container, you need to perform a rollback operation: that is, a rollback operation will be performed if it fails when you call containerd API to create a new container. If the API returns normal, but the abnormal running of the program in the container causes the container to exit, the rollback operation will not be performed.

A basic operation of the rollback operation is given as follows:

Defer func () {

If! needRollback {

Return

}

/ / rollback to old container.c.meta = & backupContainerMeta// create a new containerd container.if err: = mgr.createContainerdContainer (ctx, c); err! = nil {logrus.Errorf ("failed to rollback upgrade action:% s", err.Error ()) if err: = mgr.markStoppedAndRelease (c, nil); err! = nil {logrus.Errorf ("failed to mark container% s stop status:% s", c.ID (), err.Error ()}})

} ()

During the upgrade process, if an exception occurs, the newly created Snapshot and other related resources will be cleaned. In the rollback phase, you only need to restore the configuration of the old container, and then start a new container with the restored configuration file.

Upgrade function demonstration

Use the ubuntu image to create a new container:

$pouch run-- name test-d-t registry.hub.docker.com/library/ubuntu:14.04 top

43b75002b9a20264907441e0fe7d66030fb9acedaa9aa0fef839ccab1f9b7a8f

$pouch ps

Name ID Status Created Image Runtime

Test 43b750 Up 3 seconds 3 seconds ago registry.hub.docker.com/library/ubuntu:14.04 runc

Upgrade the image of the test container to busybox:

$pouch upgrade-name test registry.hub.docker.com/library/busybox:latest top

Test

$pouch ps

Name ID Status Created Image Runtime

Test 43b750 Up 3 seconds 34 seconds ago registry.hub.docker.com/library/busybox:latest runc

As demonstrated above, the container image is directly replaced with a new image through the upgrade API, while other configurations remain unchanged.

Summary

In the enterprise production environment, container upgrade operation is a high-frequency operation as well as container expansion and reduction operations. However, neither the current Moby community nor the Containerd community has taken the lead in implementing this function, which solves a painful point of container technology stateful service update and release in the enterprise environment. PouchContainer is now also trying to maintain close contact with its downstream dependent component services such as Containerd, so it will later give upgrade functionality back to the Containerd community to increase the functional richness of Containerd.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.