Shulou(Shulou.com)06/01 Report--

This article shows you how to use the Admission Webhook mechanism to achieve multi-cluster resource quota control, the content is concise and easy to understand, absolutely can make your eyes bright, through the detailed introduction of this article, I hope you can get something.

1 the problem to be solved

When a cluster is allocated to multiple users, it needs to use quotas to limit the use of resources, including CPU cores, memory size, GPU cards, and so on, so as to prevent resources from being exhausted by some users, resulting in unfair resource allocation.

In most cases, the native ResourceQuota mechanism of the cluster can solve the problem well. However, as the size of the cluster increases and the number of task types increases, we need to adjust the rules for quota management:

ResourceQuota is designed for a single cluster, but in fact, multi-cluster environments are often used in development / production.

Most of the tasks in the cluster are submitted through high-level resource objects such as deployment and mpijob. We hope to be able to determine the quota at the submission stage of the high-level resource object. However, ResourceQuota uses pod as the granularity when calculating resource requests, so it can not meet this requirement.

Based on the above problems, we need to carry out quota management on our own. Kubernetes provides a dynamic admission mechanism that allows us to write custom plug-ins to implement requested admission. Let's start with our quota management plan.

2 the principle of cluster dynamic admission

After the request to enter the K8s cluster is received by API server, it will be executed in the following sequence:

Authentication / authentication

Admission control (change)

Format verification

Admission control (authentication)

Persistence

The request is processed accordingly in the first four stages mentioned above, and whether it is allowed to pass or not is determined in turn. After each phase has passed, it can be persisted, that is, stored in the etcd database, thus becoming a successful request. During the admission control (change) phase, mutating admission webhook is called to modify the content in the request. In the admission control (authentication) phase, validating admission webhook is called to verify that the content of the request meets certain requirements, thus deciding whether to allow or deny the request. These webhook support extensions and can be independently developed and deployed to the cluster.

Although webhook can also check and reject requests during the admission control (change) phase, the order in which it is called cannot be guaranteed and cannot restrict other webhook from modifying the requested resources. Therefore, we can achieve the purpose of resource quota management by deploying validating admission webhook for quota verification and configuring it in the admission control (verification) phase to check the requested resources.

3 scenario 3.1 how to deploy the verification service in the cluster

Using a custom validating admission webhook in a K8s cluster requires deployment:

ValidatingWebhookConfiguration configuration (requires the cluster to enable ValidatingAdmissionWebhook), which is used to define which resource objects (pod, deployment, mpijob, etc.) are to be verified, and to provide the service callback address used to actually handle the verification. It is recommended to configure Service in the cluster to provide the address of the verification service.

The service that actually handles verification can be accessed through the address configured in ValidatingWebhookConfiguration.

In a single cluster environment, the verification service is deployed in the cluster in the form of deployment. In a multi-cluster environment, you can choose:

Virtual kubelet,cluster federation and other schemes are used to merge multiple clusters into a single cluster, which is reduced to the deployment of a single cluster scheme.

Deploy the verification service in one or more clusters in the form of deloyment, but be careful to ensure that the service is connected to each cluster network.

It should be noted that in both single-cluster and multi-cluster environments, services that handle verification require resource monitoring, which is generally implemented by a single point. Therefore, it is necessary to make a choice.

3.2 how to implement verification service 3.2.1 verification service architecture design 3.2.1.1 basic components

API server: cluster request entry, calling validating admission webhook to validate the request

API: access service interface, using the AdmissionReview data structure of the cluster convention as request and return

Quota usage service: API for requesting resource usage

Admissions: implementation of admission services, including access for different resource types such as deployment and mpijob

Resource validator: perform quota verification on resource requests

Quota adapter: interface with external quota service for validator query

Resource usage manager: resource usage manager to maintain resource usage and achieve quota judgment

Informers: monitor resources in the cluster, including deployment and mpijob, through the watch mechanism provided by K8s to maintain current resource usage

Store: stores resource usage data, which can be implemented in local memory of the service or Redis service implementation

3.2.1.2 the basic process of judging resource quota

For example, a user creates a deployment resource:

When a user creates a deployment resource, the definition needs to include an annotation with specified application group information, such as ti.cloud.tencent.com/group-id: 1, which indicates that an application is applied for using the resources in application group 1 (if there is no application group information, it is directly rejected or submitted to the default application group, such as application group 0, etc.).

The request is received by API server. Because ValidatingWebhookConfiguration is correctly configured in the cluster, during the verification phase of admission control, the API of the validating admission webhook deployed in the cluster is requested, using the structure AdmissionReviewRequest specified by K8s as the request, and the AdmissionReviewResponse structure is expected to be returned.

After receiving the request, the quota verification service enters the logic of the admission responsible for processing deployment resources, and calculates the resources that need to be newly applied or released for this request based on the action of changing the request to CREATE or UPDATE.

Extract the resources to be applied for from the spec.template.spec.containers [*] .resources.requests field of deployment, such as cpu: 2 and memory: 1Gi, expressed as apply.

Resource validator looks up quota adapter to get quota information for application group 1, such as cpu: 10 and memory: 20Gi, expressed as quota. Together with the apply obtained above, apply for resources from resource usage manager.

Resource usage manager has been monitoring the resource usage of deployment through informer monitoring and maintaining it in store. Store can use local memory, so there is no external dependency. Or use Redis as a storage medium to facilitate service level expansion.

When Resource usage manager receives a request from resource validator, you can use store to find out the resources currently occupied by application group 1, such as cpu: 8 and memory: 16Gi, expressed as usage. Check and find apply + usage

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.