Shulou(Shulou.com)06/02 Report--

Author | Shengdong Aliyun after-sales technical expert

Guide: compared with other features of K8s cluster, the automatic pull of private images may seem relatively simple. In most cases, the failure to pull the image is related to permissions. Therefore, when dealing with related issues, we often easily say: this problem is very simple, it must be a question of authority. But the reality is that we often spend more than one person's time on one problem but can't find the reason. This is mainly because we do not have a deep understanding of the principle of image pull, especially the principle of private image automatic pull. In this article, the author will lead you to discuss the relevant principles.

Sequentially, the automatic pull of private images will first go through the Ali Cloud Acr credential helper component, then through the API Server and kubelet components of the K8s cluster, and finally to the docker container runtime. But my narrative will start from back to front, from the most basic docker image pull.

Pull this little thing by mirror image

For the sake of discussion, let's imagine a scenario. Many people will use the network disk to store some files, such as photos, documents and so on. When we access files, we need to provide the account password to the network disk so that the network disk service can verify our identity. At this time, we are the owners of file resources, while the network disk acts as a resource server. The account password is used as an authentication method to ensure that only we can access our own files.

Cdn.com/6b9d7ab4225d4d10ec91b23f609d34f92f6df4fd.png ">

The scenario was simple enough, but soon we encountered a new demand: we needed to use an online photo album app. According to the normal use process, we need to download the photos of the network disk to the local, and then upload the photos to the electronic photo album. This process is quite tedious. The optimization we can think of is to let the album app directly access the network disk to get our photos, and this requires us to authorize the user name and password to the album app.

The advantages of this authorization method are obvious, but the disadvantages are also obvious: we give the user name and password of the network disk to the photo album service, and the photo album service has the ability to read and write the network disk, which is terrible from the point of view of data security. In fact, this is a general scenario that many applications will encounter. In fact, this is the same scenario for pulling private images. The image repository here, just like the network disk, is a resource server, while the container cluster is a three-party service. It needs to access the image repository to obtain images.

Understand the OAuth 2.0 protocol

The OAuth protocol is a standard solution designed to solve the above problems, and our discussion focuses on version 2.0. Instead of sending the account password directly to the three parties, this protocol uses an indirect way to achieve the same goal. As shown in the figure below, this protocol consists of six steps: three-party applications obtain user authorization, three-party applications obtain temporary Token, and three-party applications access resources.

These six steps are not easy to understand, mainly because the design of the security protocol needs to consider the provability of the protocol, so we interpret the protocol in a different way. To put it simply, this agreement actually does two things:

In the case of user authorization, the three-party application obtains the temporary access represented by the token; then the three-party application uses the token to obtain resources.

If the example of the network disk is used to illustrate, it is that the user authorizes the network disk service to create a temporary token for the photo album application, and then the photo album application uses this token to go to the network disk service to obtain the user's photos. In fact, the core difference among the variants of OAuth 2.0 lies in the first thing, which is the way users authorize resource servers.

The simplest one is suitable for situations where the three-party application itself has access to access to resources. In this case, the three-party application only needs to log in to the resource server with its own account password and apply for temporary token; when the user trusts the three-party application enough, the user directly gives the account password to the three-party application, and the three-party application uses the account password to apply to the resource server for temporary token; users through the interface provided by the resource server, logs in to the resource server and authorizes the resource server to issue token to the three-party application. It is also the safest to fully implement the OAuth 2.0 protocol. The three-party application first obtains the user authorization represented by the CAPTCHA, then uses the CAPTCHA to exchange the temporary token from the resource server, and finally uses token to access the resource.

From the above description, we can see that the resource server actually plays the roles of authentication and resource management. if the two are implemented separately, the protocol process will look like this.

Big picture of the role played by Docker

The implementation of the image repository Registry currently uses the method of "giving the account password to three-party applications". That is, assuming that the user trusts Docker enough, the user gives the account password directly to Docker, and then Docker uses the account password to apply for temporary token with the authentication server.

Understand docker login

First of all, before pulling the private image, we need to use the docker login command to log in to the image repository. The login here does not actually establish a session or other relationship with the mirror repository. There are three main things to do when logging in:

The first thing is to ask the user for the account password.

As shown in the following figure, when the login command is executed, this life will prompt for the account password, which corresponds to the first step of the big picture.

Second, docker accesses the https address of the image repository and confirms whether the interface returns the Docker-Distribution-Api-Version header field by challenging the v2 interface.

There is no corresponding step in the protocol diagram. Its function is similar to that of ping, except to confirm whether the v2 image repository is online and whether the versions match.

Third, docker uses the account password provided by the user to access the authentication server address Bearer realm returned by the Www-Authenticate header field.

If this access is successful, the authentication server will return the token in jwt format to docker, and then docker will encode the account password and save it in the .docker / docker.json file in the user's directory.

The following picture is the docker.json file after I logged in to the warehouse. This file is the only evidence that docker logs into the repository and will be read and used continuously in subsequent image repository operations. The key information auth is the base64 code of the account password.

What's going on when pulling the mirror image?

The image usually consists of two parts, one is the manifests file, which defines the metadata of the image, and the other is the mirror layer, which is the actual hierarchical file of the image. The image pull basically revolves around these two parts. Since the focus of our article is on permissions, we will only take manifests file pull as an example.

When you pull the manifests file, you basically do three things:

First, docker accesses the address of the mirror manifests directly to get the Www-Authenticate header field. This field includes the address of the authentication server Bearer realm, the image service address service, and the scope that defines the image and operation.

Next, docker accesses the Bearer realm address obtained above for authentication, and obtains a temporary token after authentication. This corresponds to the step of obtaining temporary token using the account password in the big picture of the agreement, which is read directly from the docker.json file.

Finally, use the token above to download the manifests file as an Authorization header field. This corresponds to the step of downloading the image of the protocol big image. Of course, because the image also has hierarchical files, the actual docker will also use this temporary token to download the file several times before the full image can be downloaded.

The basic function of automatic pulling of private mirror image realized by K8s

K8s clusters typically manage multiple nodes, each with its own docker environment. It is obviously inconvenient for users to log in to the image repository on the cluster node. In order to solve this problem, K8s implements the function of automatically pulling the image. The core of this function is to encode the docker.json content and pass it to Kubelet as part of the Pod definition as Secret.

Specifically, the steps are as follows:

Create a secret. The .dockerconfigjson data entry of the secret includes a base64-encoded docker.json file; create the pod, and the imagePullSecrets in the pod orchestration points to the secret;Kubelet created in the first step as the cluster controller to monitor the cluster changes. When it finds that a new pod has been created, it will obtain the definition of pod through API Server, which includes calling the secret;Kubelet referenced by imagePullSecrets to create a container and passing .dockerconfigjson to docker;. Finally, docker uses the decoded account password to pull the image, which is consistent with the method in the previous section. Advanced mode

The above functions, to some extent, solve the problem that it is not convenient for cluster nodes to log in to the image warehouse. But when we create the Pod, we still need to specify the imagePullSecrets for the Pod. K8s further optimizes the above basic functions through change admission control (Mutating Admission Control).

The contents of further optimization are as follows:

After the first step of creating secret, add default service account's reference to imagePullSecrets; Pod uses default service account by default, and service account change admission controller adds imagePullSecrets configuration to pod orchestration if default service account references imagePullSecrets. Acr credential helper implemented by Aliyun

Aliyun CCS team implemented the controller Acr credential helper on the basis of K8s. This controller allows users who use both Aliyun K8s cluster and container image service products to automatically use container images in private repositories without configuring their own account passwords.

Specifically, the controller listens for changes in acr-configuration, the configmap, and is mainly concerned with the configuration of acr-registry and watch-namespace. The former configuration is specified as the address of the image repository authorized by the temporary account, while the latter configuration management can automatically pull the namespace of the image. When the controller finds that there is a namespace that needs to be configured but is not configured, it will obtain the temporary account and password through the API of the Ali Cloud container image service.

With the temporary account password, Acr credential helper creates the corresponding Secret for the namespace and changes the default SA to reference the Secret. In this way, the functions of the controller and the K8s cluster itself automate the whole process of pulling the image from the Ali Cloud container image service by the Ali Cloud K8s cluster.

Summary

There is a difficulty and a key point in understanding the implementation of automatic pull of private image.

The difficulty is the principle of OAuth 2.0 security protocol, and the above mainly analyzes why OAuth is designed in this way; the emphasis is on the principle of cluster controller, because the whole automation process is actually the result of the cooperation of multiple controllers, including Admission control and Acr credential helper.

* * "Alibaba Yun × × icloudnative × × erverless, containers, Service Mesh and other technical areas, focus on cloud native popular technology trends, cloud native large-scale landing practice, do the best understanding of cloud native development × ×

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.