Shulou(Shulou.com)06/01 Report--

In this issue, the editor will bring you about how to use the custom controller Enhanced Statefulset based on K8s. The article is rich in content and analyzes and describes for you from a professional point of view. I hope you can get something after reading this article.

The next product to be talked about is Enhanced statefulset.

As the name implies, Enhanced statefulset is a further extension of the controller based on statefulset. It mainly solves the problem of Pod binding static IP, but also solves the limitation that statefulset does not allow multiple instances to be upgraded at the same time. In addition, it can support the migration of Pod and IP in the event of node failure.

As mentioned earlier, an important scenario of static IP is that the surrounding components that the business depends on are identified by IP as the instance, so you still need to keep the IP of the Pod instance unchanged after you go to Kubernetes. I believe many users are faced with similar problems, so let's share the principle of implementation.

We mainly support Enhanced Statefulset's Pod binding static IP by extending Enhanced Statefulset Controller, Scheduler, and CNI modules.

The management of static IP by Enhanced Statefulset Controller is mainly realized by maintaining and updating Static IP CR. When Controller receives a creation request, it first checks whether the instance to be created already has a corresponding static IP CR record, and creates a new record if it does not exist. After scheduler completes scheduling and CNI completes static IP allocation, controller listens for Pod information and updates it to Static IP CR. On the other hand, if the corresponding static IP CR record already exists when the instance is created, the Pod is deleted and rebuilt. Controller updates the information in static IP CR to Pod, and scheduler and CNI perform affinity scheduling and IP allocation according to the configuration on pod.

Load type, load name, node, IP, and Pod information are recorded in StaticIP CRD. Where the IP information is annotation on the Pod instance.

New cache: based on the original Node cache, a new staticIPNode cache is added to calculate and cache staticIPPod resource usage, number of cache IP, and Pod cpu/ memory usage.

Add predicate:

Based on the existing resources of PodFitsResourcesWithStaticIPPodPred Node, the resources occupied by staticIPPod are filtered again to achieve the purpose of resource preemption.

CheckPodAnnotationWithStaticIPPred checks whether the pod contains the specified node annotation of static ip and retains only the fit node list of the specified node results.

To sum up, the core idea is to identify the resources occupied by static IP Pod as a special resource separately, and to achieve the purpose of resource preemption by matching scheduling.

CNI mainly implements the following three functions in static IP scenarios:

Allocate according to the IP specified on Pod annotation. If there is no IP, randomly allocate an IP from the IP pool and update this record to Pod.

IP address reservation. When the Pod bound with static IP deletes and rebuilds, CNI will check the static IP CR record. If the record is not deleted, the IP address will not be released to ensure that the reconstructed Pod can get the bound IP.

In large-scale cluster scenarios, in order to improve SDN network performance, we require that CNI must use IP range mode. In this mode, the elastic Nic is bound to IP CIDR, such as 10.0.0.0 IP 24, rather than a specific one. IP migration, release and application are all in the form of CIDR.

Finally, we show how an Enhanced statefulset object binds a static IP through a creation process:

The user creates an enhanced statefulset object, and the request is first sent to API server

When enhanced statefulset controller listens to the request, it first queries whether there is a CR record corresponding to the Pod. If not, create a new CR. If there is already a CR, update the information in the CR in the new Pod.

Enhanced statefulset controller starts to create Pod

Scheduler dispatches the Pod to the corresponding node according to the affinity information of the pod. If there is no pod information, the new pod is scheduled normally. At the same time, the resource reservation logic will be triggered to ensure that the resources of the existing static IP Pod will not be occupied.

CNI views Pod static IP records. If there are no records, randomly assign IP and update IP information to Pod. If there are any records, assign them by record.

StaticIP controller listens for static IP information changes on Pod and updates this information to CR.

In addition to supporting the strong requirement of static IP, the second focus we consider is to empower Kubernetes's DevOps capabilities to business scenarios as much as possible. Native community StatefulSet does not allow multiple instances to be upgraded at the same time, mainly for the sake of some stateful applications that need to upgrade sequentially. However, the problem caused by this is that the efficiency is too low, and the group business has a certain tolerance for upgrade failure and sequence. In order to improve the upgrade efficiency, we define the MaxUnavailable parameter, which allows application instances to be upgraded in parallel, and always keeps the maximum number of unavailable instances within the limit of MaxUnavailable.

In addition, in order to ensure that the upgrade is sufficiently controllable, Enhanced Statefulset can be upgraded in batches through Partitions. After each batch upgrade is completed, the next upgrade is triggered by updating Partitions again, and if you find problems during the upgrade process, you can also carry out Rollback rollback or Paused pause.

Through these optimizations, Enhanced Statefulset has more flexibility to be compatible with native Statefulset rules and upgrade strictly according to the order of instances, ensuring the reliability of stateful services. You can also have the ability to upgrade concurrently in a more efficient way, similar to Deployment. At the same time, it can also be triggered manually in batches, basically covering most of the scenarios of the group's business.

Let's learn more about it through an example:

The user creates an Enhanced Statefulset application with 6 copies, the application from staticip-example-0 to staticip-example-5,Partitions is set to 3, and the MaxUnavailable is set to 2.

1apiVersion: jke.jdcloud.com/v1alpha1

2kind: EnhancedStatefulSet

3metadata:

4 name: staticip-example

5 annotations:

6 staticip.jke.jdcloud.com/enable: "true" # turn on static IP function

7spec:

8 serviceName: enhanced-sts-service-example

9 replicas: 6

10 selector:

11 matchLabels:

12 apps: staticip-example

13 updateStrategy:

14 rollingUpdate:

15 maxUnavailable: 2 # maximum number of unavailable, allows parallel upgrades, and tolerates replica unavailability

16 partition: all Pod created by 3 # enhanced statefulset have index, and the naming starts from 0. For example, all pod-0 pod-1 instances whose index is greater than or equal to the partitioning value are upgraded in batches by changing the partitioning value.

17 paused: false

18 podUpdatePolicy: ReCreate

19 type: RollingUpdate

20 template:

21 metadata:

22 labels:

23 apps: staticip-example

24 spec:

25 containers:

26-image: change nginx:v1 # nginx:v1 to nginx:v2 to trigger upgrade

When a user promotes an image from v1 to v2, the upgrade process is as follows:

Enhanced Statefulset Controller upgrades the three copies of staticip-example-3 to staticip-example-5 concurrently to v2, where staticip-example-4 is not available because the current value of MaxUnavailable is 2, which does not affect the continued upgrade of the application.

The user sets Partitions to 0bot enhanced statefulset controller and upgrades the remaining 3 replicas from staticip-example-0 to staticip-example-2 and to v2 version, where staticip-example-2 is not available

Then the user fixes the unavailable Pod manually, and all instances return to normal.

When performing the second step, if two instances of the upgrade in the first step are unavailable to trigger the MaxUnavailable threshold, the user will not trigger the upgrade again even if the Partitions is set to 0 in the second step.

Finally, I'll talk to you about the failover feature. Static IP brings convenience to business Kubernetes as well as problems. One of the more prominent problems is the failure migration scenario. There are several prerequisites for failover:

Static IP Pod and its bound IP need to be migrated to the same target node to ensure that the IP remains unchanged after Pod migration

As mentioned earlier, in large-scale clusters, we require that CNI must be configured in IP Range mode. In this mode, IP CIDR cannot be split to finer-grained migration, and an IP CIDR bound to a node can only be migrated to one target node. This means that all Pod bound to static IP must also be migrated to the same destination. This leads to the question of how to ensure that the target node has sufficient resources.

After the failure migration, the business wants to retain the original physical topology, virtual machine configuration and specifications to the maximum extent.

To solve these problems, our current scheme is node migration. The basic process is as follows:

When the node is lost and exceeds the tolerated time window (the user can configure the time window threshold according to the business situation), node operator will disable this node.

Node operator will create a target node with the same specification and AZ as the failed node.

Node operator migrates the IP and Pod assignments of the failed node to the new node and updates the metadata information

Delete the failed node.

The above is the editor for you to share the custom controller based on K8s Enhanced Statefulset how to use, if you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.