Shulou(Shulou.com)06/02 Report--

Although the command set of docker and kubernetes is not very complex and the background operation is relatively fast, it is still a worry for most enterprises and individual users wandering in the containerization door. Docker or not docker, that's a question,SWR service provides a minimalist containerization delivery platform for enterprises and individual users by providing interface operations, shielding the original life command line, simplifying user operation and technical threshold. We will introduce these features of SWR through a series of articles.

What I want to introduce to you today is the user 0 command line, which realizes the upload of the image and the analysis of the principle through the WEB interface.

Let's start with this most commonly used and extremely simple docker push function. Why? As we found in the process of communicating with customers, most of them have not come into contact with containerized management systems, or even mirroring, they are not familiar with the back-end operation, they have a certain demand for page operation. At present, the mainstream PaaS platforms basically support building images, creating clusters, creating applications and so on through page operations. They are constantly encapsulating the interfaces of the underlying cluster management systems (such as kubernetes) and designing a front-end page that is friendly to users under the cloud, so that as many complex back-end operations as possible can be completed with a few mouse clicks.

We can interpret this trend as the lower the cost of cloud business (including money cost and time cost), the greater the tendency to go to the cloud. Today, we support users to complete construction, deployment and other operations on the page. If you can upload and download images on the page, users can avoid back-end operations as much as possible and spend as much time and cost on business debugging as possible in the early stage of trying to cloud. Ordinary OPS personnel do not need to be familiar with docker commands, but can also download images from the private network or third-party image repositories, upload and complete the upgrade operation.

Next, we start with the image upload logic and image structure, and explain how to achieve the function of uploading images on the page.

Analysis of back-end upload Mirror flow

Our goal is to implement another way of uploading images. First of all, we need to understand what the original process of uploading images looks like.

Upload mirror layer

When docker push, the first file to be uploaded is the mirror layer file. As in the following busybox, the short ID of each line represents the sha256 value of a mirror layer, which has two mirror layers:

Upload metadata file

Because of the sequential dependency between layers, we can think of that the uploaded layer file is not enough to describe the whole image. In addition to mirror layer files, docker push also uploads a mirrored metadata file. This file mainly stores the mirror environment variables, layer structure, build information, and so on, and its sha256 value is the mirror ID. Because there are too many fields, the meaning of each field is not listed in detail here. Interested friends can use the docker inspect command to check it and refer to the official docker documentation for more information.

Upload manifest

Have you noticed that after each image is uploaded, there will be an extra line of xxx: digest: xxx size: xxx on the screen. The last line of information is printed, indicating that the last part of the image has been uploaded. This part of the data is manifest, and the long ID after digest is the sha256 value of manifest.

Manifest is mainly responsible for associating metadata files and mirror layers of mirrors. After all layers have been uploaded, it is sent to the warehouse to verify whether all physical files have been uploaded. By grabbing the package or consulting the official documents, we can see that the structure of manifest is as follows:

From the above analysis, to fully describe a mirror image, you need to store the following data:

Mirror layer

Metadata file

Manifest

Let's take a look at whether we can get this data from the image package generated by docker save.

Analysis of the structure of Image Compression package

Save the image compression package through docker save, and after unpacking it, you can find that its file structure is more orderly.

There are three files in the root directory:

In addition, there are several directories named with long ID in the package, each with the following three files:

Here, there are two common misunderstandings that need to be clarified:

Myth 1: manifest.json is manifest

Manifest describes the names of the metadata files, the sha256 values of each layer, as well as their sizes.

What is stored in manifest.json is not complete manifest information, it only records the full path name of the metadata file and the full path name of each mirror layer, and does not record the sha256 value and size of each layer.

Myth 2: the directory name of each layer is the sha256 value of the mirror layer.

In fact, the directory name is a special sha256 value calculated by combining the chain ID (chain ID) of each layer and the chain ID of the associated parent layer. This special sha256 value, which we can call v1 ID, is designed to be compatible with earlier versions (prior to 1.10) of docker images, where there may be multiple layers with the same sha256 value (such as empty layers) in a single image.

Incidentally, the above chain ID is obtained by docker daemon using recursion to calculate the sha256 of each layer and all dependent parent layers, which can effectively solve the problem of directory name duplication caused by the same layer. I will not elaborate on the specific calculation method here.

After understanding these two points, we can find that the mirror data as complete as docker push can be obtained in the image package. Among them, the image layer and metadata files can be obtained directly by decompressing, while manifest needs to be obtained by supplementing manifest.json. Next, let's take a look at how Huawei's cloud container image service implements this process.

How is the page uploaded?

Decompress and verify

When the image package is uploaded to the backend, first check the file types in the package (ordinary files, soft links, directories). After confirming that it is correct, decompress it to a temporary directory and verify the size (there is a size limit for front-end upload).

In addition, there is one type of image that needs to be filtered: the mirror package generated by the docker save image_id > image.tar command. There is no valid image repository and version number information for this kind of image, and we cannot determine which repository to attribute it to. Therefore, such an image can be considered illegal. For page uploads, the legal image archive must contain image repository and version number information (such as images generated using docker save repository:tag > image.tar).

Save entity Fil

Next, through the manifest.json in the temporary directory, find the corresponding metadata file xxxx.json and the mirror layer files in each directory to store. Before saving, check the actual image file through the sha256 value of each layer in the metadata file xxxx.json. In the process of saving, we add the sha256 value and size of each mirror layer and metadata file on the basis of manifest.json to get manifest.

There is a point to note here, layer files are generally ordinary files, but in some cases (such as the previous version of docker1.10), layer files may be soft links to another layer file in the same image package. If you want to be compatible with the old version, you need to identify this part of the special file and skip saving the physical file.

Save metadata

Finally, the mirror layer metadata list and the mirror metadata are stored in the database in the same transaction to ensure that the storage of the mirror metadata is an atomic operation, then all the mirrored data is saved. The image can be downloaded normally through docker pull.

This is just one of the features developed by Huawei Cloud Container Image Service for the purpose of optimizing user experience. We have been working to reduce the threshold and cost of cloud container technology, and promote the containerization process in the software industry. I hope interested friends can come and give us your valuable advice.

In addition, we have recently launched a continuous container delivery tool, which can quickly compile and build your source code into an image, eliminating the tedious process of local writing Dockerfile, image production, release and deployment. We will explain it in more detail later.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.