Shulou(Shulou.com)05/31 Report--

This article mainly explains "what is the method based on Kubernetes and OpenKruise". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "what is the method based on Kubernetes and OpenKruise".

1. Thinking about variable Infrastructure 1.1 variable and immutable Infrastructure in kubernetes

At a time when the cloud is becoming more and more popular, the concept of immutable infrastructure has gradually taken root in the hearts of the people. Immutable infrastructure was first proposed by Chad Fowler in 2013, and its core idea is that once an instance of any infrastructure is created, it becomes read-only, and if it needs to be modified and upgraded, it will be replaced with a new instance. Under the guidance of this concept, the consistency of running examples is achieved, so it shows unparalleled advantages in improving release efficiency, flexible scaling and upgrade rollback.

Kubernetes is an excellent platform for practicing the concept of immutable infrastructure. As the smallest unit of k8s, Pod takes on the role of application instance. Through ReplicaSet, the number of copies of Pod is controlled, and the elastic scaling of Pod is realized. When updating, Deployment controls the number of copies of the two ReplicaSet to replace the instance as a whole to achieve upgrade and rollback operations.

Let's further think about whether we need to use Pod as a completely immutable infrastructure instance. In fact, in kubernetes itself, it has provided a function to replace image to achieve the replacement of Container by changing the image field without changing the Pod. This advantage is that there is no need to re-create the Pod to achieve the upgrade, and the direct advantage is that it avoids the time of rescheduling and so on, so that the container can start quickly.

Extending from this idea, we can actually divide Pod and Container into two layers. Container is regarded as an immutable infrastructure to ensure the complete replacement of application instances, while Pod is regarded as a variable infrastructure, which can be changed dynamically, that is, variable layer.

1.2 Analysis of upgrade changes

For the types of upgrades and changes in applications, let's have a classified discussion and divide them into the following categories:

Upgrade change type specification change cpu, memory and other resource usage modification configuration change environment variable, configuration file change image change code modification after image update health check change readinessProbe, livenessProbe configuration modification other changes scheduling domain, label modification and other modifications

(swipe to see the full table)

We have done a sample survey for different types of changes, and we can see a statistical result in the following figure.

If there are more than one change in an upgrade, it is counted as multiple times.

You can see that the replacement that supports mirroring can cover about half of the upgrade changes, but there are still quite a few cases where you have to recreate the Pod. At this point, it is not very friendly. So we made a design that divides the changes to Pod into three types of Dynamic,Rebuild,Static.

Modification type modification type description modification example corresponding to change type Dynamic dynamic modification Pod unchanged, container need not be rebuilt, health check port health check change Rebuild update Pod unchanged, container needs to re-create updated image, configuration file or environment variable image change, configuration change Static static modification Pod needs to re-create modified container specification change

(swipe to see the full table)

This way of dynamic modification and in-situ update can cover more than 90% of the upgrade changes. The benefits are also obvious when the Pod remains the same.

The time of scheduling, network creation and so on is reduced.

Because most of the layers of the image of the same application are reused, the time to pull the image is greatly shortened.

Resource locking prevents resources from being preempted by other businesses and unable to run when the cluster resources are in shortage.

IP remains unchanged and is very friendly to many stateful services.

2. Customization of Kubernetes and OpenKruise 2.1 kubernetes customization

So how do you implement Dynamic and Rebuild updates? Kubernetes needs to be customized here.

Dynamic modification and customization

The dynamic modification support of liveness and readiness is relatively simple. The main modification point adds a UpdatePod function to prober_manager to determine that when the configuration of liveness or readiness changes, stop the original worker and restart the new worker. Then embed UpdatePod into kubelet's HandlePodUpdates process.

Func (m * manager) UpdatePod (pod * v1.Pod) {m.workerLock.Lock () defer m.workerLock.Unlock ()

Key: = probeKey {podUID: pod.UID} for _, c: = range pod.Spec.Containers {key.containerName = c.Name {key.probeType = readiness worker, ok: = m.workers [key] if ok {if c.ReadinessProbe = = nil {/ / readiness is empty The original worker stops worker.stop ()} else if! reflect.DeepEqual (* worker.spec, * c.ReadinessProbe) {/ / readiness configuration has changed The original worker stops worker.stop ()} if c.ReadinessProbe! = nil {if! ok | | (ok & &! reflect.DeepEqual (* worker.spec, * c.ReadinessProbe)) {/ / readiness configuration has changed Start a new worker w: = newWorker (m, readiness, pod, c) m.workers [key] = w go w.run ()}} {/ / liveness is similar to readiness. }}}

Update customization in place

Kubernetes natively supports changes to image, while modifications to env and volume are not supported. So we also support modifications to env and volume so that they can replace environment variables and configuration files. A little trick here is that we have added an ExcludedHash to calculate the various configurations in the Container, including env,volume.

Func HashContainerExcluded (container * v1.Container) uint64 {copyContainer: = container.DeepCopy () copyContainer.Resources = v1.ResourceRequirements {} copyContainer.LivenessProbe = & v1.Probe {} copyContainer.ReadinessProbe = & v1.Probe {} hash: = fnv.New32a () hashutil.DeepHashObject (hash, copyContainer) return uint64 (hash.Sum32 ())

In this way, when env,volume or image changes, it can be sensed directly. In the case of SyncPod, when calculating the computePodActions, the container is Rebuild when it is found that the relevant configuration of the container has changed.

Func (m * kubeGenericRuntimeManager) computePodActions (pod * v1.Pod, podStatus * kubecontainer.PodStatus) podActions {. For idx, container: = range pod.Spec.Containers {. If expectedHash, actualHash, changed: = containerExcludedChanged (& container, containerStatus); changed {/ / when env,volume or image is replaced, the container is rebuilt. Reason = fmt.Sprintf ("Container spec exclude resources hash changed (% d vs d).", actualHash, expectedHash) restart = true}. Message: = reason if restart {/ / add the container to the rebuilt list message = fmt.Sprintf ("% s.Container will be killed and recreated.", message) changes.ContainersToStart = append (changes.ContainersToStart, idx)}. Return changes}

Life cycle of Pod

From the completion of the scheduling of the Pod to the creation of the Running, there will be a status of the ContaienrCreating to identify the container in the creation process. In native, when image is replaced, a previous container is destroyed, and during the creation of the latter container, the Pod status is always in Running, which is prone to incorrect traffic import, and the user cannot recognize the status of the container at this time.

Therefore, we add the status of ContaienrRebuilding in ContainerStatus for in-situ update, and set the Ready Condition of Pod to False before the container is created successfully, so that the container is being rebuilt, and the application is not available during this period. With this identification, it is convenient to identify Pod status and block traffic during this period.

2.2 customization of OpenKruise

OpenKruise is an open source project of Ali, which provides a set of extended workload management and implementation beyond the Kubernetes core controller. Advanced StatefulSet, an enhanced version based on native StatefulSet, has the same default behavior as the native. In addition, it provides features such as in-situ upgrade, parallel release (maximum unavailable), release pause, and so on.

The in-place upgrade in Advanced StatefulSet is the same as the Rebuild in this article, but natively only replaces images are supported. Therefore, we customize it on the basis of OpenKruise, so that it can not only support the in-situ update of image, but also support the in-situ update of env and volume and the dynamic update of livenessProbe and readinessProbe. This is mainly judged in the shouldDoInPlaceUpdate function. There is no more code presentation here.

In addition, there is a small pit, that is, the controller-revision- hash value is added in pod to identify different versions.

[root@xxx ~] # kubectl get pod-n predictor- o yaml predictor-0 apiVersion: v1kind: Podmetadata: labels: controller-revision-hash: predictor-85f9455f6...

In general, this value should only use the hash value as the value, but the way {sts-name} + {hash} is used in OpenKruise, a small problem is that sts-name is limited because of the length of the label value.

Thank you for your reading, the above is the content of "what is the method based on Kubernetes and OpenKruise". After the study of this article, I believe you have a deeper understanding of what the method based on Kubernetes and OpenKruise is, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.