Shulou(Shulou.com)06/02 Report--

Author | Jiuzhu Aliyun technical expert

Introduction: OpenKruise is an open source automation management engine for large-scale applications of Aliyun, which features native Kubernetes Deployment/StatefulSet and other controllers, but OpenKruise provides more enhanced features such as elegant in-place upgrade, release priority / fragmentation strategy, multi-availability zone workload abstract management, unified sidecar container injection management, etc., all of which are the core competencies honed by Alibaba's super-large-scale application scenarios. These feature help us cope with more diverse deployment environments and requirements, and bring more flexible deployment and release portfolio strategies for cluster maintainers and application developers.

Currently, in Alibaba's internal cloud native environment, most applications uniformly use the capabilities of OpenKruise for Pod deployment and release management, while many industry companies and Ali Cloud customers have turned to OpenKruise as the application deployment carrier because the load such as K8s native Deployment can not fully meet their needs.

Background problem

Before introducing the new capabilities of OpenKruise, let's take a look at the publishing capabilities provided by native K8s workload:

Deployment currently supports maxUnavailable and maxSurge:

StatefulSet currently supports partition:

Other workload, such as DaemonSet, only support maxUnavailable.

The above strategies are feasible in the test environment or small scenarios, but they can not fully meet the large-scale application scenarios. For example:

First of all, Deployment does not support grayscale batch release. Do you want to verify only 20% of the Pod in grayscale upgrade? Sorry, I can't do this. Users can only set a smaller maxUnavailable to wait for it to finish, or issue an emergency pause pause. StatefulSet does support grayscale batching (partition), but currently it can only be upgraded one by one Pod. If the total number of replicas is hundreds or thousands, then a release may have to wait until dark. What's new in v0.5.0

Here we only introduce the two main functional changes of v0.5.0 version CloneSet and SidecarSet. Interested students can see the details of the version changes on Github changelog: https://github.com/openkruise/kruise/blob/master/CHANGELOG.md.

CloneSet supports maxSurge policy

In Alibaba's native cloud environment, most stateless applications are managed by CloneSet. To meet the extreme deployment requirements of very large-scale applications, we support:

Upgrade in place (Pod object unchanged, IP unchanged, volume unchanged, only upgrade container images before and after release) narrowing replicas specifies Pod to delete rich publishing strategies (streaming, grayscale batching, priority, fragmentation, etc.)

In the February version of Kruise v0.4.0, we introduced CloneSet to open source. CloneSet has attracted wide attention since it was released, and a number of well-known Internet companies have been investigating and using it.

The original version of CloneSet does not support maxSurge (expand and then release), only maxUnavailable, partition and other strategies are supported. This is not a problem for a large number of applications within Alibaba, but many community users have small-scale applications on their platforms. If they cannot be configured to expand first and then shrink later, the availability of applications may be affected during the release phase.

After receiving feedback from the community of issue # 250 # 260, we added support for the maxSurge policy in CloneSet and provided it in version 0.5.0. We would also like to thank members of the community, such as fatedier shiyan2016, for their participation and contribution and valuable suggestions. So far, CloneSet has covered all the publishing strategies of K8s native workload. The following figure builds the publishing features currently provided by CloneSet:

For the time being, we will not elaborate on the release strategy of CloneSet. We will have a special article to introduce it later. Let's just take a look at how the new maxSurge is implemented in conjunction with streaming and batch releases. Let's take a look at a few simple examples:

Set maxSurge + maxUnavailable + partition publish: apiVersion: apps.kruise.io/v1alpha1kind: CloneSet#... spec: replicas: 5 # Pod Total 5 updateStrategy: maxSurge: 20% # expand 5 * 20% = 1 Pod (rounding up) maxUnavailable: 0 # guarantee release process 5-0 = 5 Pod available partition: 3 # keep 3 old versions of Pod (only release 5-3 = 2 Pod)

When you start publishing, CloneSet will first expand one more Pod based on maxSurge, and the total number of Pod will be 6 (5 old versions and 1 new version):

$kubectl get clone demoNAME DESIRED UPDATED UPDATED_READY READY TOTAL AGEdemo 5 1 0 5 6 17m

Subsequently, CloneSet will gradually delete the Pod and update it in a new way on the premise of ensuring maxUnavailable, until it meets the partition=3, that is, the remaining 3 old versions of Pod. At this point, because the expected final state has been reached, CloneSet will delete one of the new versions of Pod, and the total number of Pod will be 5 (3 old versions and 2 new versions):

$kubectl get clone demoNAME DESIRED UPDATED UPDATED_READY READY TOTAL AGEdemo 5 2 2 5 5 17m

You can observe here for a while, and when you need to continue to publish, change the partition to 0 again. Then, CloneSet will expand one more Pod based on maxSurge again, and the total number of Pod is 6 (3 old versions, 3 new versions):

$kubectl get clone demoNAME DESIRED UPDATED UPDATED_READY READY TOTAL AGEdemo 5 3 2 5 6 17m

Subsequently, CloneSet will gradually delete and update the Pod on the premise of ensuring maxUnavailable, until it meets the partition=0, that is, all Pod are upgraded to the new version. Finally, CloneSet will delete one of the new versions of Pod, with a total of 5 Pod (5 new versions):

$kubectl get clone demoNAME DESIRED UPDATED UPDATED_READY READY TOTAL AGEdemo 5 5 5 55 17mmaxSurge with in-place upgrade:

CloneSet provides two upgrade methods: Pod in-situ upgrade and reconstruction upgrade, both of which can be released in conjunction with strategies such as maxSurge / maxUnavailable / partition.

ApiVersion: apps.kruise.io/v1alpha1kind: CloneSet#... spec: updateStrategy: type: InPlaceIfPossible maxSurge: 20%

If maxSurge,CloneSet is configured in the local upgrade mode, the Pod of the number of maxSurge will be expanded first, then the old version of Pod will be upgraded by in-place (update the image image in Pod spec), and finally the number of Pod of maxSurge will be deleted after the final state of partition is met.

In this way, it not only ensures the business availability of the publishing process, but also tries to keep the IP, volume and other information in the Pod publishing process unchanged.

SidecarSet supports volume injection merge

SidecarSet is another important feature provided by Kruise. Unlike CloneSet/StatefulSet, which manages business Pod, workload,SidecarSet is responsible for the unified management of sidecar container versions and injections in the cluster.

The new feature in v0.5.0 is to resolve the conflict of repeated definition of volume in SidecarSet and Pod when sidecar container is injected. This is also feedback from a community called issue # 254, which uses SidecarSet for log collection sidecar management and expects to be injected into all Pod in a bypass way.

For example, we need to inject a log collection sidecar container into each Pod in the cluster. But on the one hand, we can't get every application developer to add the definition of this container to their own CloneSet/Deployment, and on the other hand, even if we add it to the workload of all applications, if we want to upgrade the mirror version of this log collection container, we have to update the workload of all applications, which is too expensive!

The SidcarSet provided by OpenKruise is designed to solve the above problem. We only need to write the sidecar definition to a global SidcarSet. Regardless of whether the user deploys it in CloneSet, Deployment, StatefulSet, etc., the expanded Pod will be injected into our defined sidecar container.

Taking log collection as an example, we can first define a SidecarSet:

ApiVersion: apps.kruise.io/v1alpha1kind: SidecarSetmetadata: name: log-sidecarspec: selector: matchLabels: app-type: long-term # inject containers:-name: log-collector image: xxx:latest volumeMounts:-name: log-volume mountPath: / var/log # into all Pod with long-term tags to mount log-volume volumes to the / var/log directory Collect logs in this directory volumes:-name: log-volume # define a volume named log-volume emptyDir: {}

Here you may ask, what if each application logs out to a different directory path? Don't worry, this is what volume merge is all about.

At this time, for example, an original Pod with An expansion is as follows:

ApiVersion: v1kind: Podmetadata: labels: app-type: long-termspec: containers:-name: app image: xxx:latest volumeMounts:-name: log-volume mountPath: / app/logs # apply your own log directory volumes:-name: log-volume # define a volume called log-volume persistentVolumeClaim: claimName: pvc-xxx

Then kruise webhook injects the log sidecar container defined in SidecarSet into Pod:

ApiVersion: v1kind: Podmetadata: labels: app-type: long-termspec: containers:-name: app image: xxx:latest volumeMounts:-name: log-volume mountPath: / app/logs # apply your own log directory-name: log-collector image: xxx:latest volumeMounts:-name: log-volume mountPath: / var/log volumes:-name: log-volume # define a log directory named log-volume Volume persistentVolumeClaim: claimName: pvc-xxx

As you can see, because the log volume defined in SidecarSet and Pod is called log-volume, the volume defined in Pod will prevail when injected. For example, if the volume in the Pod uses pvc to mount the pv, then after sidecar is injected, the volume will also be mounted to the / var/log directory in the sidecar container, and then log collection can be performed.

In this way, the sidecar container is managed in the way of SidecarSet, which is not only decoupled from the application deployment and release, but also can share volume volumes with the application container to achieve log collection, monitoring and other related sidecar functions.

Summary

The upgrade of v0.5.0 mainly brings more convenient capabilities for lossless release of applications and sidecar container management.

In the future, OpenKruise will continue to make further optimization in application deployment / release capabilities. We also welcome more students to participate in the OpenKruise community to build a richer and perfect K8s application management and delivery expansion capability, which can be oriented to more large-scale, complex and extreme performance scenarios.

Course recommendation

In order that more developers can enjoy the dividends brought by Serverless, this time, we have gathered 10 + technical experts in Alibaba's Serverless field to create an open Serverless course that is most suitable for developers to get started, so that you can easily embrace the new paradigm of cloud computing-Serverless.

Click to watch the course for free: https://developer.aliyun.com/learning/roadmap/serverless

"Alibaba Cloud Native focus on micro services, Serverless, containers, Service Mesh and other technology areas, focus on cloud native popular technology trends, cloud native large-scale landing practice, to be the official account of cloud native developers."

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.