Containerized RDS: expand Kubernetes storage capacity with CSI 02/24 Update SLTechnology News&Howtos

Containerized RDS: expand Kubernetes storage capacity with CSI

2025-02-24 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Database >

Shulou(Shulou.com)06/01 Report--

Capacitive RDS series of articles:

Containerized RDS: computing "Split-Brain" under the architecture of storage separation

Containerized RDS: computing storage separation or local storage?

Containerized RDS: you need to understand how data is written "bad"

Containerized RDS:PersistentLocalVolumes and VolumeScheduling

RDS is not new. What is new is to build RDS through container technology and container orchestration technology. For financial customers, they have a strong desire to embrace Docker and Kubernetes, but availability is a prerequisite for trying new technologies. Storage is a key resource for persistent applications. It is not sexy, but it is the key for Monolithic applications to move towards Cloud-Native architecture. The Kubernetes storage subsystem is already very powerful, but it still lacks some basic features, such as support for Expand Volume (partial Storage Vendor support) and SnapShot. This article attempts to share the following from our implementation:

Problems with existing Kubernetes storage plug-in system

Container Storage Interface (CSI)

Dynamic extension of Volume in MySQL based on CSI and distributed file system

The Prospect of CSI

The noun describes:

Formerly known as Container choreography system CO. Storage provider SP. Storage plug-in interface Volume Plugin Interface storage driver Volume Driver

Container storage interface

Container Storage Interface

CSI

If there are any omissions, don't hesitate to give us advice.

Problems with existing Kubernetes storage plug-in system

There are many container orchestration systems available (hereinafter referred to as CO.), including Mesos, Swarm and Cloud Foundry in addition to Kubernetes. Take Kubernetes as an example, which supports the following storage through PersistentVolume abstraction:

GCEPersistentDisk

AWSElasticBlockStore

AzureFile

AzureDisk

FC (Fibre Channel) * *

FlexVolume

Flocker

NFS

ISCSI

RBD (Ceph Block Device)

CephFS

Cinder (OpenStack block storage)

GlusterFS

VsphereVolume

Quobyte Volumes

HostPath

VMware Photon

Portworx Volumes

ScaleIO Volumes

StorageOS

It is not rich. Kubernetes supports storage vendors (hereinafter referred to as SP.) in a plug-in way. SP. Provide your own storage driver (hereinafter referred to as Volume Driver) by implementing the Kubernetes storage plug-in interface (hereinafter referred to as Volume Plugin Interface).

VolumePlugin

PersistentVolumePlugin

DeletableVolumePlugin

ProvisionableVolumePlugin

ExpandableVolumePlugin

Provisioner

Deleter

All the above interfaces do not need to be implemented, in which VolumePlugin [1] is a must.

The architecture of the system is as follows:

This approach provides a rich list of storage support for Kubernetes, but in terms of implementation, SP. The Volume Driver code is also in the Kubernetes code repository (also known as in-tree), which poses several significant problems.

From Kubernetes's point of view:

You need to give each SP in Kubernetes. Empower them to submit code to the warehouse

Volume Driver is composed of each SP. Provided, Kubernetes developers do not know every detail, making the code difficult to maintain and test

The release rhythm of Kubernetes and everyone's SP. The rhythm of Volume Driver is not consistent with the support of SP. With the increase, the cost of communication, maintenance and testing will become higher and higher.

These SP. Volume Driver is not needed by Kubernetes itself.

From SP. From the perspective of:

Submitting a new feature or fixing bug requires submitting code to the Kubernetes repository, which, as anyone who compiles Kubernetes locally knows, is a painful process for SP. It is a completely unnecessary cost.

A unified, accepted container storage interface is becoming more and more necessary.

Container Storage Interface (CSI)

Based on these problems and challenges, CO manufacturers proposed Container Storage Interface to define container storage standards, which is independent of Kubernetes Storage SIG and promoted by Kubernetes, Mesos and Cloud Foundry. Personally, it has the following two core goals:

Provide a unified CO. And SP. The container storage interface that all follow.

Once SP. Its own Volume Driver is implemented based on CSI, which can migrate smoothly in all CO that supports CSI.

Note:Container Storage Interface definition [2].

Another side benefit is that once CO. And SP. Following CSI makes it easy to decouple Volume Driver to the outside of Kubernetes (also known as out-of-tree). Kubernetes only needs to maintain the CSI interface, no longer need to maintain SP. The specific implementation of the maintenance cost is greatly reduced.

Architecture diagram optimized by CSI:

As you can see, Kubernetes and SP. From then on, there was a clear distinction, but the matter was not that simple, to borrow the words of a great god: The performance improvement does not materialize from the air, it comes with code complexity increase. Like performance, scalability and decoupling do not emerge out of thin air. The above image can be further refined into the following image:

The obvious changes are:

Controller Plane and Kubelet no longer interact with Volume Driver directly, but introduce external-provisioner and external-attacher to complete the work.

SP. Volume Driver will be run by a separate container

In order to realize external-provisioner, external-attacher and SP. The interaction of Volume Driver introduces the gRPC protocol (red arrow).

There are some other changes:

Introduce a new object on the Kubernetes side:

CSIPersistentVolumeSource: this type of PV is provided by CSI Driver

VolumeAttachment: synchronize Attach and Dettach information

Introduce a new name:

Mount/umount:NodePublishVolume/NodeUnpublishVolume

Attach/dettach:ControllerPublishVolume/ControllerUnpublishVolume

Note:Kubernetes adapts to CSI design document [3].

It can be seen that in order to achieve this goal, it is more complex than before, but the benefits are huge, Kubernetes and SP. Developers can focus on their own business.

Even so, it is sometimes inevitable to extend existing CSI.

The Dynamically Expand VolumeKubernetes storage subsystem based on CSI and distributed file system on MySQL is already very powerful, but it still lacks some basic functions, such as supporting Expand Volume (partial Storage Vendor support) and SnapShot, especially Expand Volume, which is a necessary function, because with the change of business, the increase of capacity is inevitable. Once the capacity approaches the threshold, the cost of expanding storage capacity is too high.

However, the current situation is not optimistic. Kubernetes 1.10.2 uses CSI 0.2.0, which does not include Expand Volume interface, which means that even if the underlying storage supports expansion, Kubernetes cannot use this feature, so we need to do some hard code to achieve this feature:

Extended CSI Spec

Extended CSI Plugin

Implementation of Storage Driver based on CSI Spec

Demo

Other

Extended CSI Spec

As mentioned earlier, gRPC is introduced into CSI. I personally understand that the scenario in CSI has the following three advantages:

Define strongly typed structure based on protobuf, which is easy to read and understand

Realize remote call through stub, the programming logic is clearer

Support duplex and streaming, providing two-way interaction and real-time interaction

There are many articles about gRPC and protobuf on the Internet, so I won't repeat them here.

Through gRPC, the interaction of various components of CSI is realized. To support the expand volume interface, you need to modify the csi.proto and add the required rpc based on CSI 0.2.0. The key points are as follows: CSI Driver implements all the following APIs:

CreateVolume

DeleteVolume

ControllerPublishVolume

ControllerUnpublishVolume

ValidateVolumeCapabilities

ListVolumes

GetCapacity

ControllerGetCapabilities

RequiresFSResize

ControllerResizeVolume

This involves more complex debugging and adaptation work, as well as some other work:

Define the StorageClass corresponding to CSI, and set allowVolumeExpansion to true

Enable Feature Gates:ExpandPersistentVolumes

New Admission Control:PersistentVolumeClaimResize

……

Demo

By extending Container Storage Interface 0.2.0, we have implemented online file system expansion on Kubernetes 1.10.2. Create two types of workloads for MySQL instances:

Update and query through key values

Bulk data load data

From the image above, you can see:

Reading one: MySQL QPS is within normal fluctuation range

Reading 2: continuously loading data in bulk, the capacity of the MySQL file system is getting larger and larger.

Reading 3: in 20 minutes, online dynamic expansion of Volume and Filesystem twice, the process is efficient and smooth.

Other

The work is basically finished here, and there are a lot of changes to be made, which is not simple objectively. Without Kubernetes and CSI, it would be more difficult.

This method can be used to extend other Storage Vendor, such as FCSan, iSCSI.

The Prospect of CSI

At present, CSI has developed to 0.2.0 and 0.3.0 will be released soon.

The most popular feature in 0.3.0 is Snapshot. With this function, backup and remote disaster recovery can be realized. However, in order to achieve this function, a new Controller needs to be added to the existing Control-Plane of Kubernetes. Objectively, the complexity will be further improved.

At the same time, some in-tree Volume Driver has been migrated externally through CSI. Considering the overall release rhythm of Kubernetes and the stability of API, I think the pace will not be too fast.

Beyond that, CSI may have more work to do. Take a highly available scenario as an example. When a Node fails, CO triggers the process of Unmount- > Dettach- > Attach- > Mount. With the drift of Pod, Unmount, Dettach, Attach and Mount are created by SP with the interface defined by CSI. Its own reality, but the process of Unmount- > Dettach- > Attach- > Mount is still controlled by CO, which is not standard, but is critical to workload.

Also think of the sentence "No Silver Bullet" in.