Shulou(Shulou.com)05/31 Report--

K8s is an example of accessing storage through ceph-csi. In order to solve this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

Overview

The following analysis is that K8s is connected to ceph storage through ceph-csi (the analysis of related components of csi takes rbd as an example). It briefly introduces the K8s objects and components involved, as well as the process analysis of storage operations. Storage-related operations include storage creation, storage expansion, storage mount, storage unmount and storage deletion.

Associative link

Ceph-csi Suite Analytical Directory Navigation

Involving k8s object 1. PersistentVolume

Persistent storage volumes, cluster-level resources, represent storage volume resources and record information about the storage volume resources.

Recycling strategy

(1) retain: retention policy, PV and external storage resources still exist when PVC is deleted.

(2) delete: delete policy. When the pvc bound to pv is deleted, the PV object is deleted from the K8s cluster and the deletion of external storage resources is performed.

(3) resycle (abandoned)

Pv state transition

Available-- > bound-- > released

2. PersistentVolumeClaim

Pvc state transition

Pending-- > bound

3. StorageClass

Define the template information for creating pv, cluster-level resources, which are used to dynamically create pv.

Involving components and roles

The following is a brief introduction to the components and functions involved below, followed by a separate and detailed description of the role of each component.

1. Controller manager

PV controller

Responsible for binding and lifecycle management of pv and pvc, and create / delete underlying storage and pv objects according to requirements.

This analysis is that k8s uses ceph storage through ceph-csi. In this analysis, the operations of creating / deleting underlying storage and creating / deleting pv objects are triggered by external-provisioner, so the PV controller in this analysis is only responsible for binding and lifecycle management of pv and pvc.

AD controller

It is mainly responsible for Attach/Detach operations of storage devices (such as creating and deleting VolumeAttachment objects, mounting data volumes to specific node nodes, etc.), and updating node.Status.VolumesAttached.

This analysis is that k8s uses ceph storage through ceph-csi. In this analysis, AD controller only creates / deletes VolumeAttachment objects, but does not actually mount / unmount data volumes to / from nodes; csi-attacer components do not mount / unmount nodes, but update VolumeAttachment objects. The real node mount / unmount operations are completed by calling ceph-csi from volume manager in kubelet.

2. Kubelet

Volume manager

It mainly manages the Attach/Detach of the volume (the same as AD controller, and controls which component to do this operation through the kubelet startup parameters, which will be described in detail later), mount/umount and other operations.

This analysis is that k8s uses ceph storage through ceph-csi. In this analysis, the Attach/Detach operation of volume manager only creates / deletes the VolumeAttachment object, but does not actually mount the data volume to / unmount the node; the csi-attacer component does not mount / unmount the node, but updates the VolumeAttachment object. The real node mount / unmount operation is accomplished by calling ceph-csi when the volume manager in kubelet calls the rc.operationExecutor.MountVolume/rc.operationExecutor.UnmountDevice method.

3. Volume plugin

Expand the management ability of various storage types of volumes to realize the combination of various operation capabilities of third-party storage and k8s storage system. Call the interfaces or commands of third-party storage to provide the creation / deletion of data volumes, attach/detach, and the implementation of specific operations of mount/umount, which can be regarded as the agent of third-party storage. The creation / deletion of data volumes, attach/detach, and mount/umount operations in the previous analysis component are all done by calling volume plugin.

The subsequent detailed analysis of volume plugin, taking the operation of rbd through ceph-csi as an example.

According to the location of the source code, volume plugin is divided into in-tree and out-of-tree.

In-tree

Out-of-tree

The code is independent of K8s and is implemented by storage manufacturers. There are two implementations: csi and flexvolume.

This analysis is that k8s uses ceph storage through ceph-csi. Volume plugin in this analysis refers to ceph-csi and its related components, including ceph-csi, external-provisioner, external-attacher, external-resizer and so on.

Analysis of operation flow related to storage

The following is an analysis of the overall flow of each operation, followed by a source code analysis of each component involved.

1. Storage creation

Flow chart

Process analysis

(1) user creates pvc object

(2) external-provisioner component listens to new events of pvc, and calls csi component to create storage.

(3) csi component calls ceph to create underlying storage

(5) pv controller listens to the pvc object, looks for a suitable pv object, and binds to the pvc object.

two。 Storage expansion

Flow chart

Process analysis

(1) modify the pvc object and the storage size of the application

(2) after the modification is successful, external-resizer listens to the update event of the pvc and finds that pvc.Spec.Resources.Requests.storgage is larger than pvc.Status.Capacity.storgage, so it adjusts the csi controller side to expand the underlying storage capacity. After the expansion, update the .Spec.Capacity.storgage of the pv object.

(3) volume manager of kubelet found that pv.Spec.Capacity.storage was greater than pvc.Status.Capacity.storage during reconcile () tuning, so the csi node end was adjusted to expand the capacity, and the corresponding file system stored on dnode was expanded. After success, kubelet updates the pvc.Status.Capacity.storage.

3. Storage mount

Flow chart

The kubelet startup parameter, enable-controller-attach-detach, is set to true to enable Attach/Detach controller to perform Attach/Detach operations and to disable kubelet to perform Attach/Detach operations (default is true). In fact, the Attach/Detach operation is to create / delete VolumeAttachment objects.

(1) kubelet startup parameter-enable-controller-attach-detach=true,Attach/Detach controller for Attach/Detach operation

(2) kubelet startup parameter-enable-controller-attach-detach=false. Kubelet volume manager performs Attach/Detach operation.

Process analysis

(1) the user creates a pod with pvc mounted

(2) reconcile () in AD controller or volume manager finds that volume does not perform the attach operation, so it performs the attach operation, that is, to create a VolumeAttachment object.

(3) external-attacher component list/watch VolumeAttachement object, update VolumeAttachment.status.attached=true

(5) volume manager in kubelet obtains the value of node.Status.VolumesAttached attribute and finds that volume has been marked as attached

(6) so reconcile () in volume manager calls the NodeStageVolume and NodePublishVolume of the ceph-csi component to complete the mount.

4. Unmount storage

Flow chart

(1) AD controller

(2) volume manager

Process analysis

(1) user deletes the pod that declares pvc

(2) reconcile () in AD controller or volume manager finds that volume does not perform the dettach operation, so it performs the dettach operation, that is, to delete the VolumeAttachment object

(3) AD controller or volume manager wait for the VolumeAttachment object to be deleted successfully

(4) AD controller updates the .Status.VolumesAttached attribute value of the new node object and removes the volume marked as attached from the attribute value

(5) the volume manager in kubelet obtains the node.Status.VolumesAttached attribute value and cannot find the relevant volume information.

(6) so reconcile () in volume manager calls the NodeUnpublishVolume and NodeUnstageVolume of the ceph-csi component to unmount.

5. Delete Stora

Flow chart

Process analysis

(1) user deletes pvc object

(2) pv controller found that the pvc object bound to pv was deleted, so the status of pv was updated to released.

(3) external-provisioner watch to pv update event, and check whether the status of pv is released and whether the recycling policy is delete

(4) next, the external-provisioner component will call ceph-csi 's DeleteVolume to delete the storage.

(5) DeleteVolume method of ceph-csi component, which calls the ceph cluster command to delete the underlying storage

(6) the external-provisioner component deletes pv objects.

This is the answer to the case analysis question of K8s connected to storage through ceph-csi. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel for more related knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.