Shulou(Shulou.com)06/01 Report--

This article is to share with you about how to understand the source code of kubernetes data volume management, the editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

Overview

Volume is a very important part of K8s, which is mainly used to store some system or business data produced by pod in K8s. K8s provides the logic of volume management in kubelet

Source code analysis

The first is the kubelet startup method

Func main () {s: = options.NewKubeletServer () s.AddFlags (pflag.CommandLine) flag.InitFlags () logs.InitLogs () defer logs.FlushLogs () verflag.PrintAndExitIfRequested () if err: = app.Run (s, nil); err! = nil {fmt.Fprintf (os.Stderr, "error:% v\ n", err) os.Exit (1)}}

It is easy to find that the run method contains all the important information of kubelet

Func run (s * options.KubeletServer, kubeDeps * kubelet.KubeletDeps) (err error) {/ / configuration verification. If kubeDeps = = nil {... KubeDeps, err = UnsecuredKubeletDeps (s).} / / initialize managers such as cAdvisor and containerManager. If err: = RunKubelet (& s.KubeletConfiguration, kubeDeps, s.RunOnce, standaloneMode); err! = nil {return err}...}

There are two important methods related to volume management

UnsecuredKubeletDeps: will initialize docker client, network management plug-in, data management plug-in and other core components of the system, because it is not convenient to open to the outside, so named unsecure. What we need to pay attention to is its initialization of volume plugin.

Func UnsecuredKubeletDeps (s * options.KubeletServer) (* kubelet.KubeletDeps, error) {... Return & kubelet.KubeletDeps {Auth: nil, CAdvisorInterface: nil, Cloud: nil, ContainerManager: nil, DockerClient: dockerClient, KubeClient: nil ExternalKubeClient: nil, Mounter: mounter, NetworkPlugins: ProbeNetworkPlugins (s.NetworkPluginDir, s.CNIConfDir, s.CNIBinDir), OOMAdjuster: oom.NewOOMAdjuster (), OSInterface: kubecontainer.RealOS {} Writer: writer, VolumePlugins: ProbeVolumePlugins (s.VolumePluginDir), TLSOptions: tlsOptions,}, nil}

When initializing volume plugin, VolumePluginDir is passed as the path to the custom plugin. The default path is * * / usr/libexec/kubernetes/kubelet-plugins/volume/exec/**

Func ProbeVolumePlugins (pluginDir string) [] volume.VolumePlugin {allPlugins: = [] volume.VolumePlugin {} allPlugins = append (allPlugins, aws_ebs.ProbeVolumePlugins ()...) AllPlugins = append (allPlugins, empty_dir.ProbeVolumePlugins ()...) AllPlugins = append (allPlugins, gce_pd.ProbeVolumePlugins ()...) AllPlugins = append (allPlugins, git_repo.ProbeVolumePlugins ()...) AllPlugins = append (allPlugins, host_path.ProbeVolumePlugins (volume.VolumeConfig {})...) AllPlugins = append (allPlugins, nfs.ProbeVolumePlugins (volume.VolumeConfig {})...) AllPlugins = append (allPlugins, secret.ProbeVolumePlugins ()...) AllPlugins = append (allPlugins, iscsi.ProbeVolumePlugins ()...) AllPlugins = append (allPlugins, glusterfs.ProbeVolumePlugins ()...) AllPlugins = append (allPlugins, rbd.ProbeVolumePlugins ()...) AllPlugins = append (allPlugins, cinder.ProbeVolumePlugins ()...) AllPlugins = append (allPlugins, quobyte.ProbeVolumePlugins ()...) AllPlugins = append (allPlugins, cephfs.ProbeVolumePlugins ()...) AllPlugins = append (allPlugins, downwardapi.ProbeVolumePlugins ()...) AllPlugins = append (allPlugins, fc.ProbeVolumePlugins ()...) AllPlugins = append (allPlugins, flocker.ProbeVolumePlugins ()...) AllPlugins = append (allPlugins, flexvolume.ProbeVolumePlugins (pluginDir)...) AllPlugins = append (allPlugins, azure_file.ProbeVolumePlugins ()...) AllPlugins = append (allPlugins, configmap.ProbeVolumePlugins ()...) AllPlugins = append (allPlugins, vsphere_volume.ProbeVolumePlugins ()...) AllPlugins = append (allPlugins, azure_dd.ProbeVolumePlugins ()...) AllPlugins = append (allPlugins, photon_pd.ProbeVolumePlugins ()...) AllPlugins = append (allPlugins, projected.ProbeVolumePlugins ()...) AllPlugins = append (allPlugins, portworx.ProbeVolumePlugins ()...) AllPlugins = append (allPlugins, scaleio.ProbeVolumePlugins ()...) Return allPlugins}

You can observe that one of the many plug-ins is called flexvolume, and only this plug-in has the parameter pluginDir, indicating that only this plug-in supports a custom implementation. How kubelet interacts with these plug-ins and what interfaces these plug-ins provide, you still need to read the code.

RunKubelet: this is the way to start kubelet services, and the most important functions are hidden in startKubelet

Func RunKubelet (kubeCfg * componentconfig.KubeletConfiguration, kubeDeps * kubelet.KubeletDeps, runOnce bool, standaloneMode bool) error {/ / initializes the initiator. If runOnce {if _, err: = k.RunOnce (podCfg.Updates ()) Err! = nil {return fmt.Errorf ("runonce failed:% v", err)} glog.Infof ("Started kubelet% s as runonce", version.Get (). String ())} else {startKubelet (k, podCfg, kubeCfg) KubeDeps) glog.Infof ("Started kubelet% s", version.Get () .String ())} return nil}

StartKubelet consists of two links.

Func startKubelet (k kubelet.KubeletBootstrap, podCfg * config.PodConfig, kubeCfg * componentconfig.KubeletConfiguration, kubeDeps * kubelet.KubeletDeps) {/ / synchronize pod information go wait.Until (func () {k.Run (podCfg.Updates ())}, 0 Wait.NeverStop) / / start kubelet service if kubeCfg.EnableServer {go wait.Until (func () {k.ListenAndServe (net.ParseIP (kubeCfg.Address), uint (kubeCfg.Port), kubeDeps.TLSOptions, kubeDeps.Auth, kubeCfg.EnableDebuggingHandlers, kubeCfg.EnableContentionProfiling)}, 0 Wait.NeverStop)} if kubeCfg.ReadOnlyPort > 0 {go wait.Until (func () {k.ListenAndServeReadOnly (net.ParseIP (kubeCfg.Address), uint (kubeCfg.ReadOnlyPort))}, 0, wait.NeverStop)}}

The Run method that tracks synchronous pod information traces back to this code

Func (kl * Kubelet) Run (updates & 2} log () {echo-ne $* > & 1} ismounted () {MOUNT= `findmnt-n ${MNTPATH} 2 > / dev/null | cut-d''- f1`if ["${MOUNT}" = "${MNTPATH}"] Then echo "1" else echo "0" fi} getdevice () {VOLUMEID=$ (echo ${JSON_PARAMS} | jq-r '.volumeID') VG=$ (echo ${JSON_PARAMS} | jq-r '.volumegroup') # LVM substitutes-with-- VOLUMEID= `echo $VOLUMEID | sed sed attach attach () {JSON_PARAMS=$1 SIZE=$ (echo $1 | jq -) {JSON_PARAMS=$1 SIZE=$ (echo $1 | jq -) R '.size') DMDEV=$ (getdevice) if [!-b "${DMDEV}"] Then err "{\" status\ ":\" Failure\ ",\" message\ ":\" Volume ${VOLUMEID} does not exist\ "}" exit 1 fi log "{\" status\ ":\" Success\ " \ "device\":\ "${DMDEV}\"} "exit 0} detach () {log" {\ "status\":\ "Success\"} "exit 0} waitforattach () {shift attach $*} domountdevice ( ) {MNTPATH=$1 DMDEV=$2 FSTYPE=$ (echo $3 | jq-r'. ["kubernetes.io/fsType"]') if [!-b "${DMDEV}"] Then err "{\" status\ ":\" Failure\ ",\" message\ ":\" ${DMDEV} does not exist\ "}" exit 1 fi if [$(ismounted)-eq 1] Then log "{\" status\ ":\" Success\ "}" exit 0 fi VOLFSTYPE= `blkid-o udev ${DMDEV} 2 > / dev/null | grep "ID_FS_TYPE" | cut-d "="-f2`if ["${VOLFSTYPE}" = ""] Then mkfs-t ${FSTYPE} ${DMDEV} > / dev/null 2 > & 1 if [$?-ne 0] Then err "{\" status\ ":\" Failure\ " \ "message\":\ "Failed to create fs ${FSTYPE} on device ${DMDEV}\"} "exit 1 fi fi mkdir-p ${MNTPATH} & > / dev/null mount ${DMDEV} ${MNTPATH} & > / dev/null if [$?-ne 0] Then err "{\" status\ ":\" Failure\ " \ "message\":\ "Failed to mountdevice ${DMDEV} at ${MNTPATH}\"} "exit 1 fi log" {\ "status\":\ "Success\"} "exit 0} unmountdevice () {MNTPATH=$1 if [!-d ${MNTPATH}] Then log "{\" status\ ":\" Success\ "}" exit 0 fi if [$(ismounted)-eq 0] Then log "{\" status\ ":\" Success\ "}" exit 0 fi umount ${MNTPATH} & > / dev/null if [$?-ne 0] Then err "{\" status\ ":\" Failed\ " \ "message\":\ "Failed to unmount volume at ${MNTPATH}\"} "exit 1 fi log" {\ "status\":\ "Success\"} "exit 0} isattached () {log" {\ "status\":\ "Success\" \ "attached\": true} "exit 0} op=$1 if [" $op "=" init "] Then log "{\" status\ ":\" Success\ "}" exit 0 fi if [$#-lt 2]; then usage fi shift case "$op" in attach) attach $* Detach) detach $*;; waitforattach) waitforattach $*;; mountdevice) domountdevice $* Unmountdevice) unmountdevice $*;; isattached) isattached $* *) log "{\" status\ ":\" Not supported\ "}" exit 0 esac exit 1

It is worth noting why there are two mount operations, one mountdevice and one mount. What do you do respectively?

In fact, the volume management method provided by K8s is that a volume can be mounted by multiple pod. If a device needs to be the volume of multiple pod, it needs to be mounted multiple times. But device can only be mounted once. So, K8s uses mountdevice to mount device to a global directory, which can then be mounted to pod's volume directory multiple times. In this way, multiple pod mounts of the same volume can be completed

AttachVolume: calling attach

DetachVolume: calling detach

MountVolume: calling mountdevice,mount

UnmountVolume: calling unmount

UnmountDevice: calling umountdevice

If there is no binding relationship between the expected state of volume and pod, detach volume, and perform unmount operations on pod and volume

If there is a binding relationship between the expected state of volume and pod, attach volume, and perform mount operations on pod and volume

DesiredStateOfWorld: expected usage of volume by pod, or expected state for short. When pod.yaml customizes the volume and successfully submits it, the expected status has been determined

ActualStateOfWorld: in practice, the use of voluem by pod, referred to as the actual status. The actual status is the result of background thread monitoring of kubelet

Constantly synchronize the pod information of apiserver, and update the volume status synchronously according to the newly added or deleted pod

Start the service to listen for controller manager requests. Among them, controller manager can assist kubelet to manage volume, and users can choose to disable controller manager management.

Only if you understand the code of volume manager, can you be familiar with using the volume plugin it provides or implementing a custom flex volume plugin. The above code is based on k8s v1.6.6

The above is how to understand the source code of kubernetes data volume management. Xiaobian believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.