Shulou(Shulou.com)06/01 Report--

This article is about how to understand Operator and Operator Framework in Kubernetes API. I think it is very practical, so I share it with you. I hope you can get something after reading this article.

1. Operator outlines the basic concepts

First of all, introduce the basic concepts involved in this article.

CRD (Custom Resource Definition): allows users to customize Kubernetes resources, which is a type

CR (Custom Resourse): a concrete example of CRD

Webhook: it is essentially a HTTP callback that is registered with apiserver. When a specific apiserver event occurs, the registered webhook is queried and the corresponding message is forwarded.

According to the type of processing, it can generally be divided into two categories: one may modify the incoming object, and the other, called mutating webhook;, will read only the incoming object, called validating webhook.

Work queue: the core component of controller. It monitors resource changes within the cluster and stores related objects, including its actions and key, such as a Create action of Pod, as an event in the queue.

Controller: it will cycle through the above work queues, pushing the cluster state to the expected state according to their respective logic. Different controller processes different types. For example, replicaset controller focuses on the number of replicas and handles some Pod-related events.

Operator:operator is a set of mechanisms for describing, deploying and managing kubernetes applications. In terms of implementation, it can be understood as CRD cooperating with optional webhook and controller to implement user business logic, that is, operator = CRD + webhook + controller.

Common working modes of operator

Workflow:

The user creates a custom resource (CRD)

Apiserver forwards the request of the pass to webhook according to a list of CRD registered by itself.

Webhook generally completes the default setting and parameter verification of the CRD. After the webhook is processed, the corresponding CR is written to the database and returned to the user

At the same time, controller monitors the custom resource in the background and handles the special operations associated with the custom resource according to the business logic

The above processing generally causes state changes within the cluster, and controller monitors these associated changes and records these changes in the state of the CRD.

Here is a brief introduction from High-Level, which will be re-combed with the case later.

II. Overview of operator framework actual combat operator framework

Before we begin, let's introduce operator framework. In fact, it provides users with the framework of webhook and controller, and its main significance is to help developers shield some general underlying details, so that developers no longer need to implement message notification triggering, failure re-queuing, etc., just pay attention to the operation and maintenance logic implementation of the managed application.

There are two main operator framework types: kubebuilder and operator-sdk.

In fact, there is no essential difference between the two, their core is the use of the official controller-tools and controller-runtime. However, the details are slightly different, such as kubebuilder has more sophisticated testing and deployment and code generation scaffolding, while operator-sdk has better support for upper-level operations such as ansible operator.

Kuberbuildere actual combat

The actual combat here is kuberbuilder. The case is SidercarSet under Aliyun's kruise project, which is open to the public.

The function of SidercarSet is to insert sidecar containers (also known as auxiliary containers) into the Pod. For example, you can insert some monitoring and log collection to enrich the functions of the Pod, and then update the SidercarSet according to the status of the insertion and the status of the Pod to record the status of these auxiliary containers.

Step 1: initialize

Action: create a new gitlab project and run "kubebuilder init-domain=kruise.io".

Parameter interpretation: domain specifies the Group domain name for subsequent registration of CRD objects.

Effect interpretation: pull dependent code base, generate code framework, generate Makefile/Dockerfile and other tool files.

Let's take a look at the specific content generated by it _ (the actual generated file has been partially deleted for demonstration purposes):

You can confirm the specific content in detail during the actual combat.

Step 2: create API

Operation: run "kubebuilder create api-group apps-version v1alpha1-kind SidecarSet-namespace=false"

In fact, you will not only create API, that is, CRD, but also generate a framework for Controller.

Parameter interpretation:-group plus the previous domian, that is, the Group: apps.kruise.io of this CRD

Version is generally divided into three types, according to community standards:

V1alpha1: this api is unstable, CRD may be obsolete, fields may be adjusted at any time, do not rely on

V1beta1: api has been stabilized, backward compatibility is guaranteed, and features may be adjusted

V1: api and features are stable

Kind: the type of this CRD, similar to the concept of Service native to the community

Namespaced: whether this CRD is globally unique or namespace unique, similar to node and Pod.

Its parameters can be divided into two categories. Group, version and kind basically correspond to the three important components of CRD meta-information. Here are some common standards, which you can refer to when you actually use them. Namespaced is used to specify whether the CRD we just created is globally unique (such as node) or namespace unique (such as Pod). False is used here, which specifies that SidecarSet is globally unique.

Effect interpretation:

The framework for CRD and controller is generated, and the code needs to be manually populated later.

The actual results are shown in the following figure:

We focus on the blue font part of the picture. Sidecarset_types.go is the place where CRD is defined and needs to be populated. Sidecarset_controller.go is used to define controller, which also needs to be populated.

Step 3: populate CRD

1. The generated CRD is located in "pkg/apis/apps/v1alpha1/sidecarset_types.go" and usually requires the following two operations:

(1) adjust comments

Code generator relies on comments to generate code, so you sometimes need to adjust comments. The following is a list of notes that need to be adjusted for SidecarSet in this actual combat:

+ genclient:nonNamespaced: generate non-namespace objects

+ kubebuilder:subresource:status: generate status child resources

+ kubebuilder:printcolumn:name= "MATCHED", type='integer',JSONPath= ".status.matchedPods", description= "xxx": kubectl get sidecarset: related to subsequent shows.

(2) fill in the field

Populating fields is an important part of making the user's CRD actually effective and meaningful.

SidecarSetSpec: populating CRD description information

SidecarSetStatus: populates the CRD status information.

two。 Just fill in and run make to regenerate the code.

It should be noted that R & D personnel do not need to participate in the underlying implementation of CRD's grpc interface, codec and other controller.

The actual fill is shown in the following figure:

The function of SidecarSet is to inject Sidecar into Pod. To accomplish this, we define two main pieces of information in SidecarSetSpec (left): one is the Selector; used to select which Pod needs to be injected, and the other is the Containers that defines the Sidecar container.

State information is defined in SidecarSetStatus (right). MatchedPods reflects how much Pod,UpdatedPods is actually matched by the SidecarSet reflects how much has been injected, and ReadyPods reflects how much Pod is working properly.

For complete content, you can refer to this document.

Step 4: generate webhook framework

1. To generate the mutating webhook, run:

Kubebuilder alpha webhook-group apps-version v1alpha1-kind SidecarSet-type=mutating-operations=create

"kubebuilder alpha webhook-group core-version v1-kind Pod-type=mutating-operations=create"

two。 To generate the validating webhook, run:

Kubebuilder alpha webhook-group apps-version v1alpha1-kind SidecarSet-type=validating-operations=create,update

Parameter interpretation:

Group/kind describes the resource objects that need to be processed

Type describes what type of framework needs to be generated

Operations describes what operations you focus on on the resource object.

Effect interpretation:

The webhook framework is generated, and the code needs to be manually populated later

The actual result is shown in the following figure:

We executed three commands to generate three different handler that need to be populated (the blue font section above). Forget about it here and explain it in detail in the next fill operation.

Step 5: populate webhook

The generated webhook handler is located at:

Pkg/webhook/default_server/sidecarset/mutating/xxx_handler.go

Pkg/webhook/default_server/sidecarset/validating/xxx_handler.go

Pkg/webhook/default_server/pod/mutating/xxx_handler.go

Generally speaking, the parts that need to be rewritten and filled include the following two parts:

Whether you need to inject K8s client or not: depends on whether other resources are needed in addition to the incoming CRD. In this actual combat, we should not only focus on SidecarSet, but also inject Pod, so we need to inject K8s client.

The key way to populate webhook: mutatingSidecarSetFn or validatingSidecarSetFn. Since the pointer to the resource object to be manipulated has been passed in, the hook work can be completed by directly adjusting the properties of the object.

Let's take a look at the actual filling results.

Because in the fourth step, we define three: sidecarset mutating, sidecarset mutaing, and pod mutating.

First, let's take a look at the sidecarset mutating on the left side of the figure. First, setDefaultSidecarSet sets the default value, which is also the most common thing mutaing does.

Validating on the right side of the above image is also a very standard, and it also verifies some fields of SidecarSet.

The pod mutaing is not shown here, which is a little different. Instead of setting the default value, the mutaingSidecarSetFn gets the value of setDefaultSidecarSet and injects it into Pod.

Step 6: populate controller

The generated controller framework is located in pkg/controller/sidecarset/sidecarset_controller.go. There are three main points that need to be modified:

Modify permission comments. The framework will automatically generate comments such as / / + kuberbuilder:rbac;groups=apps,resources=deployments/status,verbs=get;update;path, which we can modify according to our own needs, which will eventually generate rbac rules.

Increase the logic of joining the team. The default code framework populates the queuing logic of CRD itself (for example, all additions and deletions of SidecarSet objects will be added to the work queue). If you need the trigger mechanism of associated resource objects (such as SidecarSet also needs to pay attention to the changes of Pod), you need to manually add its queuing logic.

Populate the business logic. Modify the Reconcile function to cycle through the work queue. The Reconcile function mainly completes two parts: "completing the business logic according to Spec" and "feedback the business logic results back to status". It is important to note that if the Reconcile function returns err in error, it will rejoin the queue by default.

Let's take a look at the filling result of SidecarSet's Controller:

In the addPod, the SidecarSet corresponding to the Pod is first retrieved and queued for Reconcile processing.

After the SidercarSet is taken out by Reconcile, the matching Pod is selected according to the Selector. Finally, the status of the cluster is calculated based on the current status information of the Pod, and then backfilled into the status of the CRD.

III. The work flow of SidecarSet

Finally, let's rearrange the workflow of SidecarSet so that we can understand how operator works.

The user creates a SidecarSet

After webhook receives the SidecarSet, it performs default settings and configuration item verification. After these two operations are completed, the actual storage is completed and returned to the user.

The user creates a Pod

Webhook will take back the corresponding SidecarSet and take the container out of it and inject it into the Pod, so the Pod has just entered the library with the sidecar.

Controller polls constantly in the background to see the status changes of the cluster. The injection in step 4 will trigger the enlistment of SidecarSet, and controller will increase the UpdatedPods of SidecarSet by 1.

The above is a simple function implementation in the early stage of SidecarSet.

Let's add one more question here. In general, controller completes business logic and status updates in webhook, but this is not certain, and one of them may not be necessary. In the above example, the main business logic is done by webhook without the involvement of controller.

IV. Summary

This is the end of the main content, here is a brief summary for you:

Operator is a set of mechanism for CRD to expand user's business logic under Kubernetes system with optional webhook and controller.

Kubebuilder is an official and standardized Operator framework with high recognition in the community.

According to the above practical steps, fill in the user-defined code, you can easily implement an Operator.

The above is how to understand Operator and Operator Framework in Kubernetes API. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.