Shulou(Shulou.com)05/31 Report--

This article introduces the relevant knowledge of "how to apply Kubelet Bootstrap Checkpoint". In the operation of actual cases, many people will encounter such a dilemma. Next, let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

What is Kubelet Bootstrap Checkpoint?

Kubelet Bootstrap Checkpoint is a kubelet built-in module for kubelet to back up and restore a specific Pods.

Kubelet Bootstrap Checkpoint is a Checkpoint-to-file system mechanism for Pods with Annotation:node.kubernetes.io/bootstrap-checkpoint=true on current Node.

When kubelet restarts, it checks the checkpoint files corresponding to each Pods in the checkpoint directory, loads all checkpoint files, converts them to Pod Object, and then starts these Pods.

Kubelet Bootstrap Checkpoint application scenario

It looks a bit like the way static pod is used: based on the description files of Pod in a directory, kubelet monitors these files and decides whether to delete, create, and update the corresponding Pods based on whether the files have been changed or not. Or maybe it's a bit like the DaemonSet usage scenario.

The biggest difference is that Kubelet Bootstrap Checkpoint is the checkpoint of a specific Pods. If the Pods is changed or created through API, the latest Pod data will be written to the checkpoint file corresponding to Pod. The checkpoint file name format corresponding to Pod is Pod_UID.yaml, and the content stored is the complete Yaml format of Pod API Object, including Status.

Main application scenarios of Kubelet Bootstrap Checkpoint:

Self-hosted-kubernetes is used to upgrade and maintain the apiserver,controller-manager,scheduler,kubelet,etcd,Adds-on components hosted by K8s. For more information about self-hosted-kubernetes, please refer to self-hosted-kubernetes design-proposals,bootkube, kubeadm upgrade and so on. This is also the main motivation for community design of this feature. The following figure is the schematic diagram of bootkube.

So, is there any benefit for users to add Annotation:node.kubernetes.io/bootstrap-checkpoint=true to Pod to provide bootstrap checkpoint to the Pod when deploying a normal application?

For users, if apiserver can be accessed properly, then bootstrap checkpoint is really useless, because there is already Pods API Object information in etcd, and checkpoint seems superfluous. If there is an inconsistency between the checkpoint file and the data in the etcd, it will lead to the problem that the Pod is recovered through checkpoint and then rebuilt according to the Object Info in etcd.

However, for some special resident Agent on Node, such as cmdb agent, you need to report the status of Node regularly and run on Node in DaemonSet Pod mode. If the Node system restarts and cannot communicate with apiserver for a long time (for example, apiserver upgrade failure), if the Node system restarts and cannot communicate with apiserver for a long time (for example, apiserver upgrade failure), DaemonSet Pod cannot be run on Node, so the cmdb agent on this Node cannot report information properly. In this case, if we set the corresponding Annotation and enable Kubelet Bootstrap Checkpoint for the DaemonSet Pod, then kubelet can restore the previously backed up Pods from the local checkpoint file without relying on apiserver.

Therefore, it is valuable to use Bootstrap Checkpoint for some key user components on per-node.

How to enable Kubelet Bootstrap Checkpoint

Bootstrap-checkpoint-path is configured in the Kubelet startup parameter, which defaults to "", which means the default Disable.

Add Annotation:node.kubernetes.io/bootstrap-checkpoint=true to the Pods that needs Bootstrap Checkpoint.

Bootstrap Checkpoint working mechanism

When kubelet starts, it is checked in NerMainKubelet that-- bootstrap-checkpoint-path is not empty, and if not, a checkpointManager is created.

Create or change Pod

When the user submits the request to create Pod, it is scheduled by scheduler, and finally dispatched by kubelet to this node, and the creation process of Pod is started by kubelet.

If checkpointManager is not empty, kubelet will check that Pod has Annotation:node.kubernetes.io/bootstrap-checkpoint=true,kubelet and traverse all Pods when HandlePodAddtions. Before dispatchWorker creates Pod, PodManager will call API checkpoint.WritePod to write Pods that satisfies Annotation to their corresponding checkpoint file (Pod_UID.yaml).

Similarly, when the Pod Spec changes, the kubelet traverses all the Pods through the HandlePodUpdates, and the PodManager calls the checkpoint.WritePod interface to write the latest contents of the Pods that meet the Annotation to their corresponding checkpoint files.

If Error occurs in checkpoint.WritePod, it can be seen in the kubelet log, but no process exception is thrown, that is, the Pod continues to be created, but the checkpoint fails.

Delete Pod

When the user submits a request to delete Pod, kubelet traverses all Pods through HandlePodRemove, and PodManager calls API checkpoint.DeletePod to delete the checkpoint file corresponding to Pod.

If the checkpoint file corresponding to Pod does not exist, there will be no exception return, and there should be no exception return.

If the checkpoint file corresponding to Pod exists, but deletion fails, there will be kubelet Error logs, but the process will not fail.

Note that instead of checking whether the Annotation of Pod meets the criteria, it attempts to delete the file name of the individual format for all Pods. Why not check the Annotation first? Won't the performance be higher in this way? Imagine a scenario where Pod's Checkpoint Annotation is deleted when it is changed, and his checkpoint file is left behind. After that, the Pod is deleted, and when the kubelet is restarted, the deleted Pod is also restored from the checkpoint, which is bad. Of course, soon the Pod will know from the apiserver synchronization that it has been deleted, and then kubelet will delete the Pod again.

Kubelet restart

When a cold restart occurs in kubelet, it will first check whether the bootstrap-checkpoint-path is configured. If so, it will call checkpoint.LoadPods to check all the Pod_UID.yaml files in the configured directory and check the CheckSum through the FNV Hash algorithm.

After the check is passed, the contents of the checkpoint yaml file are converted into Pod API Object, and then these Pods objects are passed to Kubelet.syncLoopIteration through the channel of type kubetypes.PodUpdate, and finally dispatch is given to Kubelet podWorkers to create the corresponding Pods instance.

Bootstrap Checkpoint workflow

The code of Bootstrap Checkpoint is very simple, and there is not much. Here is a summary diagram of the corresponding code flow.

Other considerations

Currently, Bootstrap Checkpoint only Checkpoint the specific Pods of this node, and does not include the Checkpoint of other Kubernetes Object.

What's more, it is not Checkpoint for kubelet memory data. These are not what it wants to do, let alone what it should do. The more places the cluster state is stored, the more problems it will have.

That's all for the content of "how to use Kubelet Bootstrap Checkpoint". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.