Shulou(Shulou.com)05/31 Report--

This article mainly explains "what is the use of ImageApparate image", interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn "what's the use of ImageApparate mirroring?"

Background

In the environment where the business has generally been containerized, different business scenarios have different requirements for container startup. In offline computing and some online service scenarios that need to rapidly increase computing resources (scaling groups), there are often higher requirements for container startup speed.

In the whole cycle of container startup, the image pull time often takes up 70% or more. According to statistics, due to the large container image of an offline computing business, it takes up to 40 minutes to expand the capacity of thousands of Pod each time. Image distribution has become a major obstacle to the rapid elastic scaling of containers.

ImageApparate (Phantom)

To solve this problem, Tencent Cloud CCS TKE team developed the next generation image distribution solution ImageApparate (Phantom), which increases the speed of large-scale image distribution by 5-10 times.

To deal with the problems caused by the existing Docker download image mode, the discussion of the new community solution mainly focuses on the delayed loading of image data (Lazy-Pull) and the design of the new image format is no longer based on the layer as the minimum unit, but the chuck or the file itself in the image.

However, OCI V2 is still a long way from us. How do we deal with this kind of scenario?

Back to the problem itself, the current interaction logic between OCI V1 and the container runtime needs to download the full image before running the container, but how much content in the image will be used when the container starts and runs? this paper FAST'16 calculates the amount of data that some common official images in DockerHub need to read after their use and startup, and concludes that only 6.4% of the content needs to be read on average. That is to say, most of the contents in the image may not be needed at all in the whole lifecycle of the container, so if we only load 6% of the data, we can greatly reduce the pull time of the image, thus accelerating the startup speed of the container. This also provides a theoretical premise for subsequent optimization.

Therefore, the focus of reducing the container startup time is on the acquisition of the container rootfs, that is, the container image.

Based on this premise, TCR launched the ImageApparate (Phantom) container image acceleration service under the framework of compatibility with OCI V1. First of all, it is concluded that at 200 nodes, the image content accounts for 5% to 10% of the total image size. As mentioned above, compared with the traditional way of downloading all images, ImageApparate has a 5-10 times increase in the total startup time of the container. Moreover, this test does not only focus on the creation time of the container, but also continues to test the overall time from the start of the container to the time that the business process can provide services:

Sequential reading of 500MB large files tests the time from the start of the container to the completion of sequential reading of 500MB files

Random reading of 1000 small files was tested, including the time from the start of the container to the completion of reading 1000 4k-16k.

Executing the python program tests includes the time after loading the Python interpreter after booting from the container and executing a simple python code

Performing a gcc compilation test includes the time after executing gcc to compile a simple C code after booting from the container and running it.

The problem of the traditional Mode of ImageApparate Scheme Design

Since the release of Docker, great changes have taken place in the field of cloud computing, and traditional virtual machines have been gradually replaced by containers. Docker adheres to the concept of Build and Ship And Run to complete the design of container runtime and container images, leading the entire container industry. However, with the passage of time, the Ship And Run of the container has gradually exposed some problems in the face of a wide range of user requirements.

Traditional container startup and image download are as follows:

Access the image warehouse service to obtain permission authentication and obtain the image storage address

Download all mirror layers through the network access image storage address and decompress them

Use the federated file system to mount all layers as rootfs based on the mirrored layer information, and create and start the container on this file system

The design of container image has been in use since the release of Docker, and has become a de facto standard, that is, OCI V1, which we use now, uses a hierarchical design to greatly reduce space consumption, and uses all kinds of federated file systems (Aufs, Overlayfs) to jointly mount each layer to form a complete RootFS read-only root file system. The container runtime write operation will be on the top read and write layer of the federated file system. It's a very delicate design.

However, developers and users have a never-ending pursuit of speed. With the widespread popularity of cloud in business, in order to give full play to the flexibility of resources on the cloud, users often need newly expanded computing nodes that can use containerized computing power as quickly as possible (container startup service can accept traffic). At this time, this new node needs to download all the layers of the container image, greatly slowing down the container startup speed. In this scenario, the hierarchical design of container images is not fully utilized and completely invalidated.

Some issues about the OCI V1 container image format have also been discussed centrally in the community. The current tar package as the mirror layer distribution format of OCI V1 mainly has the following problems:

Content redundancy between different layers

There is no file-based addressing access capability, and you need to unpack all of them before you can access them.

No ability to unpack concurrently

Using whiteout to process file deletion and conversion in different storage types leads to inefficient decompression

TCR-Apparate OCI products

The goal of our design is to face the production level and support both image acceleration mode and normal mode on the node. in order to decouple from the normal OCI V1 image storage, we have developed the image additional storage IAS (ImageAttachStorage) combined with the external layer type (Foreign Layer) in the image Manifest, which can accelerate the creation, upload and download of the image under the OCI V1 semantics and inherit the original image permissions at the same time. The accelerated mirror Manifest index is stored in the storage of the mirror repository itself in the form of OCI artifacts.

In terms of image format, in order to support demand loading and overcome some of the shortcomings before the tar format, ImageApparate uses a read-only file system instead of the tar format. The read-only file system not only solves the ability of addressing files in the mirror layer, but also has the performance of becoming a reliable Rootfs. ImageApparate still uses a hierarchical design to specify the attachment storage address directly in the external layer of Manifest, and the additional storage layer IAS can be mounted on demand when downloading the image.

After the user enables the image acceleration feature and sets relevant rules, the ImageApparate after push image runs the following process in the background:

Users are mirrored to the TCR repository with any client (including Docker) that meets the OCI V1 interface standard.

TCR's mirroring service writes user data to the back-end storage of the image repository itself, usually COS object storage.

TCR's mirroring service checks the image acceleration rules. If the rules are met, a Webhook notification is sent to the Apparate-client component to request the image format to be changed.

After receiving the notification, the Apparate-client component writes the COS data to the IAS and uses a specific algorithm to convert each Layer of this image one by one into a Layer format that supports ImageApparate mount.

Therefore, for TCR users, it is only necessary to define the rules to mark which images need to be accelerated, while there is no change in the way CI/CD is used, and the original development pattern is naturally inherited.

Mirror attached storage IAS (ImageAttachStorage)

As the name implies, the narrow image attached storage IAS is the data storage address in addition to its own image back-end storage. IAS can use either the same object storage as the image repository, or NFS or Lustre. In addition to the storage address, the mirror attached storage in Apparate also includes a set of plug-in interfaces (compatible with Posix) and the layout (Layout) in the mirror layer IAS. Each directory in IAS represents a Layer, where content-based addressing (Content Addressable) is still used to reuse the same layer of content. Read-only file system files contain all the content in this original layer, and the entire directory tree can be obtained at any time by loading the metadata index. Currently, Apparate uses Tencent Cloud CFS high-performance version as an implementation of IAS. CFS with high throughput and low latency is very suitable for image download scenarios.

The image local cache is implemented by different IAS additional storage plug-ins themselves. The current CFS implementation uses the FScache framework as the local cache to automatically cache part of the data on the remote storage that has been accessed by page. According to the local cache capability of the current disk, it can effectively improve the performance and stability of repeated access to image data.

Runtime implementation

The IAS additional storage plug-in currently used by ImageApparate on nodes is called Apparate-snapshotter and is implemented through the proxy-snapshotter capability of containerd.

Apparate-snapshotter is mainly responsible for parsing the IAS information recorded in the mirror layer to get another data storage address. Then Apparate-snapshotter will load remote data in the data storage service and provide access to the Posix entry locally.

For example, in the CFS scenario, the remote data will be mount locally, and the mount point will be used as the entry point for subsequent local access. When remote data is needed, the snapshotter or kernel provides the ability to load on demand.

Read-only image format

Read-only is a critical attribute for mirrored file systems that support Lazy-Pull, because read-only file systems do not need to consider fragmentation and garbage collection caused by data writes and deletions, and can optimize the distribution of blocks and indexes in advance when creating a file system, which can greatly improve the read performance of the file system.

The current read-only file system supported by IAS also adds an alphabetical sort-based directory item index (directory index), which can greatly speed up the Lookup operation of directory items.

How ImageApparate uses to create acceleration components in TCR

Currently, ImageApparate needs to be enabled in whitelist for alpha function in TCR. To enable the acceleration component, you need to select the high-performance version of the corresponding CFS. Please make sure that this version of CFS is available in the region.

Create acceleration rules

Create an acceleration rule, and only the matching images or Tag in the rule will be accelerated automatically. After pushing the image to TCR, you can see that the image matching the acceleration rule will generate an OCI artifact with the suffix-apparate.

Enable acceleration function on TKE cluster side

Enable the image acceleration configuration when creating the TCR plug-in in the TKE cluster, and then label the nodes in the cluster that need to be accelerated with kubectl label node xxx cloud.tencent.com/apparate=true. The image of the Pod in the cluster can still use the original image name (such as the above test/nginx:1.9). The acceleration plug-in supports automatically selecting accelerated images for mounting. If the mirror has been accelerated, you can see that the image field of Pod in the TKE cluster has been replaced with test/nginx:1.9-apparate.

At this point, I believe you have a deeper understanding of "what is the use of ImageApparate mirroring"? you might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.