Shulou(Shulou.com)05/31 Report--

Today, I will talk to you about the detailed explanation of the Kubernetes storage system, which may not be well understood by many people. in order to make you understand better, the editor has summarized the following for you. I hope you can get something according to this article.

The Origin of Volume, PV, PVC and StorageClass

Let's start with a question: why did you introduce the concept of Volume?

"

The answer is simple: in order to persist data, the life cycle of the data does not die with the demise of the container.

"

Before introducing Kubernetes Volume, let's review that there are two common ways to use Docker Volume,Docker Volume.

In this way, Docker manages part of the host file system, which is located in the / var/lib/docker/volumes directory by default. Since the specified data volume is not created at the time of creation, docker creates default data volume itself. In this way, bind mounts can mount the files in the container to any directory of the host.

With Docker Volume, why did Kubernetes come up with his own Volume? Google's originality?

"

The answer is no. The concepts of Kubernetes Volume and Docker Volume are similar, but there are differences. The life cycle of Kubernetes Volume and Pod is the same, but not related to the life cycle of the container. When the container terminates or restarts, the data in the Volume is not lost. The Volume is cleaned only when the Pod is deleted. And whether the data is lost depends on the specific type of Volume. For example, Volume data of type emptyDir will be lost, while data of persistent type will not be lost. In addition, Kubernetes offers nearly 20 Volume types.

"

Now that we have the Volume of Kubernetes, we can fill in the Volume is field in the Yaml orchestration file, as shown in the following nfs:

....

Volumes:

-name: static-nfs

Nfs:

Server: 12.18.17.240

Path: / nfs/data/static

If you use ceph as the storage plug-in, you can define it in the orchestration file:

Volumes:

-name: ceph-vol

Cephfs:

Monitors:

-12.18.17.241purl 6789

-12.18.17.242 purl 6789

User: admin

SecretRef:

Name: ceph-secret

ReadOnly: true

Of course, as long as it is the data volume type that Kubernetes has implemented, you can define it and use it directly in the Yaml orchestration file as above.

Seeing that 80% of the work is actually done here, why design an unnecessary PV? This question will be shelved for a moment and will be explained later.

Before explaining why you should design a superfluous PV PVC, let's take a look at what is PV PVC?

"

PV is a description of persistent storage volumes.

"

PV is usually a kind of data volume created in the cluster by operation and maintenance personnel in advance and waiting for use. As follows:

ApiVersion: v1

Kind: PersistentVolume

Metadata:

Name: nfs

Spec:

Capacity:

Storage: 10Gi

AccessModes:

-ReadWriteMany

Nfs:

Server: 10.244.1.4

Path: "/ nfs"

"

PVC describes the properties of persistent storage, such as size, read and write permissions, and so on.

"

PVC is typically created by developers, as follows:

ApiVersion: v1

Kind: PersistentVolumeClaim

Metadata:

Name: nfs

Spec:

AccessModes:

-ReadWriteMany

Resources:

Requests:

Storage: 10Gi

The user-created PV PVC must be bound before it can be utilized. The premise of PV PVC binding is that the declared field size and permissions in spec in PV must meet the requirements of PVC.

Once the binding is successful, it can be defined and used in the Pod Yaml orchestration file. As follows:

ApiVersion: v1

Kind: Pod

Metadata:

Labels:

Role: web

Spec:

Containers:

-name: web

Image: nginx

Ports:

-name: web

ContainerPort: 80

VolumeMounts:

-name: nfs

MountPath: "/ usr/share/nginx/html"

Volumes:

-name: nfs

PersistentVolumeClaim:

ClaimName: nfs

When we look at this, we also think that just PV adds an extra layer of abstraction to Volume, which is not much better than declaring Volume directly in Yaml. When you think about it, why can we define Volume directly in Yaml? Because Kubernetes has helped us implement this Volume type, if we have our own storage type that is not implemented in Kubernetes, there is no way to define Volume directly in the Yaml orchestration file. At this time, the object-oriented design of PV PVC shows its value. This is also a problem often encountered in the field of software development, open source software can not meet the requirements, but also does not provide an extensible interface, no choice but to rebuild the wheel.

We often encounter such a problem in the development process. After declaring a PVC in Pod, we find that the Pod cannot be scheduled successfully because the PVC is not bound to the appropriate PV. At this time, the operation and maintenance staff is required to create a PV, followed by the successful scheduling of the Pod. PV PVC was introduced just now, and their creation process is manual. If thousands of PV are needed in the cluster, won't the operation and maintenance staff be exhausted? In practice, this approach doesn't work at all. So Kubernetes provides us with a mechanism for automatically creating PV, Dynamic Provisioning. Before introducing this automatic creation mechanism, let's take a look at Static Provisioning. What is Static Provisioning? The way to manually create PV PVC just now is Static Provisioning. You can declare StorageClass in the PV PVC orchestration file. If there is no declaration, it defaults to "." The specific interaction process is as follows:

Static allocation process

First of all, it is up to the cluster administrator to plan in advance how users in the cluster will use storage. It will pre-allocate some storage, that is, create some PV; in advance, and then when users submit their own storage requirements (PVC), the relevant components within Kubernetes will help it bind the PVC PV. Finally, when pod uses storage, you can find the corresponding PV through PVC, and it can be used. The deficiency is also very clear. First of all, it is tedious, and then the operation and maintenance staff can not predict the real storage needs of the developers. For example, the operation and maintenance personnel have created multiple 100Gi PV storage, but in the actual development process, developers can only use 10Gi, which results in a waste of resources. Of course, Kubernetes also provides us with a better way to use it, that is, what is Dynamic Provisioning?

"

Dynamic Provisioning contains the parameter information needed to create some kind of PV, similar to a template for creating a PV. The specific interaction process is as follows:

"

The controller in the Kubernetes cluster will dynamically generate the PV needed by the user by combining the information of PVC and StorageClass. After binding the PVC PV, pod can use PV. Through the StorageClass configuration to generate the storage template needed for storage, and then dynamically create PV objects according to the needs of users, so as to achieve distribution according to needs, which not only does not increase the difficulty of users, but also liberates the operation and maintenance work of cluster administrators.

Dynamic PV usage

As mentioned above in Dynamic Provisioning, operators no longer pre-allocate PV, but just create a template file, which is StorageClass. Take NFS as an example to illustrate the whole process of using dynamic PV.

Install NFS Service # install nfs

Yum-y install nfs-utils rpcbind

# Boot self-boot

Systemctl enable rpcbind nfs-server

# configure nfs file

Echo "/ nfs/data * (rw,no_root_squash,sync)" > / etc/exports

Deploy setup program apiVersion: v1

Kind: ServiceAccount

Metadata:

Name: nfs-provisioner

-

Kind: ClusterRole

ApiVersion: rbac.authorization.k8s.io/v1

Metadata:

Name: nfs-provisioner-runner

Rules:

-apiGroups: ["]

Resources: ["persistentvolumes"]

Verbs: ["get", "list", "watch", "create", "delete"]

-apiGroups: ["]

Resources: ["persistentvolumeclaims"]

Verbs: ["get", "list", "watch", "update"]

-apiGroups: ["storage.k8s.io"]

Resources: ["storageclasses"]

Verbs: ["get", "list", "watch"]

-apiGroups: ["]

Resources: ["events"]

Verbs: ["watch", "create", "update", "patch"]

-apiGroups: ["]

Resources: ["services", "endpoints"]

Verbs: ["get", "create", "list", "watch", "update"]

-apiGroups: ["extensions"]

Resources: ["podsecuritypolicies"]

ResourceNames: ["nfs-provisioner"]

Verbs: ["use"]

-

Kind: ClusterRoleBinding

ApiVersion: rbac.authorization.k8s.io/v1

Metadata:

Name: run-nfs-provisioner

Subjects:

-kind: ServiceAccount

Name: nfs-provisioner

Namespace: logging

RoleRef:

Kind: ClusterRole

Name: nfs-provisioner-runner

ApiGroup: rbac.authorization.k8s.io

-

Kind: Deployment

ApiVersion: apps/v1

Metadata:

Name: nfs-client-provisioner

Spec:

Selector:

MatchLabels:

App: nfs-client-provisioner

Replicas: 1

Strategy:

Type: Recreate

Template:

Metadata:

Labels:

App: nfs-client-provisioner

Spec:

ServiceAccount: nfs-provisioner

Containers:

-name: nfs-client-provisioner

Image: quay.io/external_storage/nfs-client-provisioner:latest

ImagePullPolicy: IfNotPresent

VolumeMounts:

-name: nfs-client

MountPath: / persistentvolumes

Env:

-name: PROVISIONER_NAME

Value: fuseim.pri/ifs

-name: NFS_SERVER

Value: 12.18.7.20

-name: NFS_PATH

Value: / nfs/data

Volumes:

-name: nfs-client

Nfs:

Server: 12.18.7.20

Path: / nfs/data

Create StorageClass template apiVersion: storage.k8s.io/v1

Kind: StorageClass

Metadata:

Name: nfs-storage

Provisioner: fuseim.pri/ifs

ReclaimPolicy: Retain

These parameters are some of the detailed parameters that need to be specified when creating a store through Kubernetes. Users do not need to care about these parameters, as in this case provisioner refers to the provisioning program that uses nfs. ReclaimPolicy refers to the dynamically created PV. What should be done with this PV when the user is finished and the Pod and PVC are deleted? what we write here is Retain, which means that when the user's pod PVC is deleted, the PV will be retained.

After submitting the template file, users only need to define PVC in the Pod yaml file to automatically create PV and PVC. ApiVersion: apps/v1

Kind: StatefulSet

Metadata:

Name: es

Spec:

.

Template:

Metadata:

Labels:

App: elasticsearch

Spec:

.

InitContainers:

.

Containers:

-name: elasticsearch

Image: docker.elastic.co/elasticsearch/elasticsearch:7.6.2

.

VolumeClaimTemplates:

-metadata:

Name: data

Labels:

App: elasticsearch

Spec:

AccessModes: ["ReadWriteOnce"]

StorageClassName: nfs-storage

Resources:

Requests:

Storage: 50Gi`

Capacity: size of the storage object

AccessModes: this is also what users need to care about, that is, how to use this PV. It can be used in three ways: ReadWriteOnce is single node read-write access; ReadOnlyMany is multiple node read-only access, a common way of data sharing; ReadWriteMany is read-write access on multiple node

StorageClassName:StorageClassName is a field that we have to specify when we talk about dynamic Provisioning, that is, we have to specify which template file to use to generate PV.

Kubernetes storage architecture PV Controller: responsible for PV PVC binding, lifecycle management, and Provision Delete operation of data volumes according to requirements AD Controller: responsible for Attach Detach operation of storage devices, mounting devices to target node Volume Manager: managing volume Mount Unmount operations, volume device formatting and operations mounted to some common directories Volume Plugins: it is mainly the implementation of all the above mount functions. PV Controller, AD Controller and Volume Manager are mainly called for operation, while the specific operation is implemented by Volume Plugins. According to the location of the source code, Volume Plugins can be divided into two categories: In-Tree and Out-of-Tree: In-Tree indicates that the source code is placed inside the Kubernetes (common NFS, cephfs, etc.), and released, managed and iterated together with Kubernetes. The disadvantages are slow iteration speed and poor flexibility. The Volume Plugins code of Out-of-Tree is independent of Kubernetes and is implemented by storage providers. At present, there are mainly two implementation mechanisms of Flexvolume CSI, which can implement different storage plug-ins Scheduler according to the storage type: realize the scheduling ability of Pod, and will do storage-related scheduling dynamic PV interaction process Kubernetes mount Volume process according to some storage-related definitions. Users create a PodPV Controller containing PVC and observe the ApiServer. If it finds that a PVC has been created but is still unbound, it will try to bind a PV and PVC to Provision, that is, to create a Volume from a specific storage medium on the remote end, and to create a PV object in the cluster Then bind the PV and PVC to Scheduler for multi-dimensional consideration, schedule the Pod to an appropriate NodeKubelet continuously watch APIServer whether there is a Pod to be dispatched to the current node Pod dispatched to a node, and the PV defined by it has not been mounted (Attach), then AD Controller will call VolumePlugin to mount the remote Volume to the device in the target node (/ dev/vdb) When Volum Manager finds that a Pod has been dispatched to its own node and Volume has finished mounting, it will perform the mount operation, mount the local device (that is, / dev/vdb) to Pod in a subdirectory on the node to start the container, and map the Volume that has been mounted locally to the container. After reading the above, do you have any further understanding of the detailed understanding of Kubernetes storage system? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.