Shulou(Shulou.com)06/02 Report--

VI. Principle of shared memory

For applications that are stateful or need to persist data,    Kubernetes not only needs to mount the directory in the container to the host directory or empDir temporary storage volume, but also needs more reliable storage to store the important data generated by the application, so that the previous data can still be used after the container application is rebuilt. In order to shield the details of the underlying storage implementation, make it easy for users to use, and make it easy for administrators to manage, kubernetes has introduced two resource objects, persistentVolume and persistentVolumeCliam, to manage storage since version v1.0.

   persistentVolume (PV) is an abstraction of the underlying network shared storage, which defines shared storage as a kind of "resource". For example, a node is also a resource that can be consumed by container applications. PV is created and configured by administrators, and it is directly related to the specific implementation of shared storage, such as shared storage provided by GlusterFS, iSCSI, RBD or GEC/AWS public cloud. It is interfaced with shared storage through plug-in mechanism for application access and use.    persistentVolumeCliam (PVC) is a user's "application" for storage resources. Just as Pod "consumes" node resources, PVC "consumes" PV resources. PVC can request specific storage space and access modes.

   uses PVC to "apply" to a certain amount of storage space is still not enough to meet the various needs of the application for storage devices. Usually applications have different requirements for the characteristics and performance of storage devices. Including higher requirements such as read and write speed, concurrency performance, and data redundancy, kubernetes introduced a new resource object, storageClass, from v1.4 to mark the characteristics and performance of storage resources. By v1.6, storageClass and dynamic resource application mechanisms have been improved to enable on-demand creation of storage volumes.

Then the following editor will explain in detail the shared storage management mechanisms such as PV, PVC, storageClass and dynamic resource provisioning.

1. Detailed introduction of PV

As a storage resource,    PV mainly includes the settings of key information such as storage capacity, access mode, storage type, recycling policy, back-end storage type and so on. Take the following configuration as an example:

ApiVersion: v1kind: PersistentVolumemetadata: name: pv1spec: capacity: storage: 5Gi accessModes:-ReadWriteOnce persistentVolumeReclaimPolicy: Recycle storageClassName: slow nfs: path: / tmp server: 172.17.0.2

The above example creates a 5G space with an access mode of ReadWriteOnce, a storage type of "slow" (here requires the system to have created a storageClass resource object named slow), a recycling policy of "recycle", and a backend storage type of nfs (and sets the IP and path of NFS server)

The types of PV supported by Kubernetes are as follows:

PersistentDisk provided by   GCEPersistentDisk # GEC Public Cloud

ElasticBlockStore provided by   AWSElasticBlockStore # AWS Public Cloud

File provided by   AzureFile # Azure Public Cloud

Disk provided by   AzureDisk # AzureDisk

  FC (Fibre Channel)

  Flexvolume

  Flocker

  NFS # Network File system

  iSCSI

  RBD (Ceph Block Device) # Ceph storage block

  CephFS

  Cinder (OpenStack block storage) # openStack Cinder Block Storage

  Glusterfs

  VsphereVolume

  Quobyte Volumes

  HostPath # host directory for stand-alone testing only

  Portworx Volumes

  ScaleIO Volumes

  StorageOS

(1) key configuration parameters of PV

① storage capacity (capacity)

  describes the capabilities of storage devices. Currently, only storage space settings (storage:xxx) are supported.

② access Mode (Access Modes)

  sets the access mode for PV, which is used to describe the access rights of user applications to storage resources. The access mode is as follows:

   ReadWriteOnce: read and write permissions, and can only be mounted by a single node

   ReadOnlyMany: read-only permission, which can be mounted by multiple node

   ReadWriteMany: read and write permissions, which can be mounted by multiple node

Note: PV may support multiple access modes, but PV can only use one access mode when mounting, and multiple access modes cannot take effect at the same time.

There are different access modes for different types of PV, and you need to match them in the definition of PV:

③ storage category (class)

  PV can set the category it stores and specify the name of a storageClass resource object through the storageClassName parameter. A PV with a specific category can only be bound to a PVC that requests that category. A PV that does not have a category set can only bind to a PVC that does not request any category. StorageClass resource objects will show their power in the following dynamic resource provisioning.

④ recovery Policy (Reclaim Policy)

  currently supports the following three recycling strategies:

   retention (Retain): data retention, which requires manual processing

   reclaim space (Recycle): simple operation of purging files (such as performing rm-rf / xx/*)

   deletion (Delete): the back-end storage connected to PV completes the deletion of volume (such as internal volume cleanup of AWS EBS, GCE PD, Azure Disk, OpenStack Cinder, etc.)

Note: currently, only NFS and hostPath support "Recycle", while AWS EBS, GCE PD, Azure Disk, and OpenStack Cinder support "Delete".

(2) the life cycle of PV

  A PV may be in one of the following four nodes during the lifecycle:

   Available: available state, also bound to a PVC

   Bound: already bound to a PVC

   Released: the bound PVC has been deleted and the resource has been released, but not reclaimed by the cluster

   Failed: automatic resource recovery failed

(3) PV mount parameters

When   mounts a PV to a node, it may need to set additional mount parameters according to the characteristics of the backend storage. This can be achieved by setting an annotation named "volume.beta.kubernetes.io/mount/options" in the definition of PV. The following editor sets the mount parameters through a gcePersistentDisk as an example:

ApiVersion: "v1" kind: PersistentVolumemetadata: name: gce-disk-1 annotations: volume.beta.kubernetes.io/mount/options: discardsepc: capacity: storage: 10Gi accessModes:-ReadWriteOnce gcePersistentDisk: fsType: ext4 pdName: gce-disk-12. Detailed introduction of PVC

We understand the definition of PV above   , so we will use it here, and we need PVC at this point. PVC, as a user's demand for storage resources, mainly includes storage space application and access mode. PV settings for information such as selection criteria and storage categories.

  next, the editor also uses an example to introduce how to define PVC to use PV:

Kind: PersistentVolumeClaimapiVersion: v1metadata: name: myclaimsepc: accessModes:-ReadWriteOnce resources: requests: storage: 8Gi storageClassName: slow selector: matchLabels: release: "stable" matchExpressions:-{key: environment, operator: In, values: [dev]}

  requests 8 GB of space, access mode is "ReadWriteOnce", PV selection criteria include tags labeled "release=stable" and contain conditions of "environment in dev", and storage category is slow (already exists in the system).

The key configurations of PVC in   are described in detail:

   Resource request (resources): only storage space size requests are supported

   Resource request (resources): only storage space size requests are supported

Selection criteria of    PV: filter the available PV through label selector, the system will find the appropriate PV according to the tag, and bind PV to PVC

   storage category: PVC can set the category of back-end storage required at definition to reduce dependence on back-end storage features. Only the PV with the class set can be filtered out by the system and bound to the PVC.

  the editor explains in detail whether to set class requirements for PVC:

   is divided into two situations:

DefaultStorageClass is enabled on     system

     if defaultStorageClass is enabled but the default storageClass does not exist in the system, it is equivalent to not enabling defaultStorageClass

     if defaultStorageClass is enabled and there is a default storageClass in the system, the system automatically creates a PV for PVC (using the default defaultStorageClass) and binds them

DefaultStorageClass is not enabled on     system

     if the storageClass is set to "" or the storageClass field is not set, then you can only select the PV with which the storageclass is not set to match and bind it.

The way for   to set the default storageClass is to define an annotation "storageclass.kubernetes.io/is-default-class=true" on storageClass, but you cannot set annotation for multiple storageClass. If this is not unique, the system cannot create a corresponding PV for PVC.

Considerations for using PV to bind to PVC:

Both  PV and PVC are restricted by namespace. Only PV and PVC in the same namespace can be bound, and only pod under the same namespace can mount PVC.

 when both selector and storageClassName are set in the PVC definition, PV and PVC can be bound only if both conditions are met.

3. The life cycle of PV and PVC

PV can be regarded as an available storage resource, and PVC is a request for a storage resource. The relationship between PV and PVC is shown in the following figure:

Next, let's introduce them one by one from the figure:

(1) Resource supply (provisioning)

  Kubernetes supports two resource provisioning modes: static mode and dynamic mode. The result of resource provisioning is the creation of a PV.

   static mode: the cluster administrator manually creates many PV and needs to set the characteristics of the back-end storage when defining the PV.

   dynamic mode: instead of manually creating the PV, the cluster administrator describes the back-end storage through the settings of the storageClass, marking it as a "type". At this point, PVC is required to declare the type of storage, and the system will automatically complete the binding of PV created in PVC.

(2) Resource binding (binding)

  after the user defines the PVC, the system will select an existing PV that meets the PVC requirements according to the PVC's request for storage resources. Once found, the PV will be bound to the user-defined PVC, and then the user's application can use this PVC. If there is no PV,PVC in the system that meets the PVC requirements, it will remain in the pending state indefinitely until the system administrator creates a PV that meets its requirements. Once PV is bound to a PVC, it is monopolized by that PVC and cannot be bound to other PVC. In this way, it is easy to waste resources. If you are using dynamic mode, the system will automatically create a PV and bind to PVC after finding a suitable storageClass for PVC.

(3) Resource usage (Using)

  Pod uses the definition of volume to mount PVC to a path within the container for use. The type of Volume is "persistentVolumeClaim", which is explained in more detail in a later example. After a PVC is mounted on the container application, it can be used exclusively for continuous use. However, multiple pod can mount the same PVC.

(4) Resource release

  when the user has finished using the storage resource, the user can delete the PVC, and the PV bound to the PVC will be marked as "released", but cannot be bound to other PVC immediately. Data previously written by the PVC may remain on the storage device, and the PV can be used again only after it has been cleared.

(5) Resource recovery

  for PV, the administrator can set a recycling policy to set what to do with legacy data after the bound PVC releases resources. Only when the storage space of PV is reclaimed can it be bound and used by the new PVC.

Finally, the editor of   uses two pictures to introduce the principles of static resource supply mode and dynamic resource supply mode, laying the groundwork for the following section:

The principles of PV and PVC in   static Mode

Principles of storageClass, PV and PVC in   dynamic mode

4. Detailed introduction of StorageClass (1) introduction and simple definition of StorageClass

As an abstract definition of storage resources,    StorageClass shields the details of back-end storage from the PVC applications set by users. On the one hand, it reduces users' attention to the details of storage resources, on the other hand, it also reduces the manual work of administrators to manage PV. The system automatically completes the creation and binding of PV, and realizes the dynamic supply of resources.

   's definition of StorageClass mainly includes the name, the provider of the back-end storage and the relevant parameter configuration of the back-end storage. Once StorageClass is created, it cannot be modified. If you need to modify it, you can only delete the previously created StorageClass rebuild. Here is an example of defining StorageClass:

Kind: storageClassapiVersion: storage.k8s.io/v1metadata: name: standardprovisioner: kubernetes.io/aws-ebsparameters: type: gp2

The above example defines a StorageClass named "standard", the provider is "aws-ebs", and its parameter is gp2.

There are two key parameters in the definition of StorageClass:

 provisioner (provider): describes the provider of storage resources, which can also be thought of as a back-end storage driver. At present, provisioner starts with "Kubernetes.io/".

 parameters (parameters): parameter settings of backend resource providers. Different provisioner includes different parameter settings

(2) several common provisioner definitions of StorageClass:

① AWS EBS Storage Volum

Kind: storageClassapiVersion: storage.k8s.io/v1metadata: name: slowprovisioner: kubernetes.io/aws-ebsparameters: type: io1 zone: us-east-1d iopsPerGB: "10"

Parameter description:

 type: optional: io1, gp2, sc1, st1. Default is gp2.

 zone:AWS zone name

 iopPerGB: for volume of type io1 only, which means the number of Icano operations per G per second

 encrypted: whether to encrypt

 kmsKeyId: Amazon Resource Name when encrypting

② GCE PD Storage Volum

Kind: storageClassapiVersion: storage.k8s.io/v1metadata: name: slowprovisioner: kubernetes.io/gce-pdparameters: type: pd-standard zone: us-centrall-a

Parameter description:

 type: optional: pd-standard, pd-ssd. Default is pd-standard.

 zone:GCE zone name

③ GlusterFS Storage Volum

ApiVersion: storage.k8s.io/v1kind: storageClassmetadata: name: slowprovisioner: kubernetes.io/glusterfsparameters: resturl: "http://127.0.0.1:8081" clusterid: 44ad5as1fd5ae1fde1fwe1d5we1d5 restauthenabled:" true "restuser:" admin "secreNamespace:" default "secreName:" heketi-sercret "gidMin:" 40000 "gidMax:" 50000 "volumetype:" replicate:3 "

Parameter description:

 resturl:Gluster REST Service (Heketi) URL address, which is used to automate the setup of GlusterFSvolume

 restauthenabled: access to Gluster REST services to enable security

 restuser: the user name to access the Gluster REST service

 secretNamespace and secretName: the resource object name of the Secret that holds the password for accessing the Gluster REST service

Cluster ID of  clustered:GlusterFS

The scope of the GID for  gidMin and gidMax:storageClass, which is used to dynamically create the GID set by PV when provisioning resources

Volume type setting of  volumetype:GlusterFS, for example: replicate:3 represents 3 copies of replicate type

④ OpenStack Cinder Storage Volum

Kind: StorageClassapiVersion: storage.k8s.io/v1metadata: name: goldprovisioner: kubernetes.io/cinderparameters: type: fast availability: nova

Parameter description:

Volumetype of  type:cinder, which is empty by default

 availability:availability zone, empty by default

(3) set the default storageClass

For    to set a default storageClass on the system, you first need to start an admission controller named "DefaultStorageClass", that is, add: DefaultStorageClass to-- admission-control in the command parameter of kube-apiserver.

Example:-- admission-control= "xxx,xxx,xxx, DefaultStorageClass"

Kind: StorageClassapiVersion: storage.k8s.io/v1metadata: name: gold annotations: storageclass.beta.kubernetes.io/is-default-class= "true" provisioner: kubernetes.io/gce-pdparameters: type: pd-ssd

Then create the storageClass, which we can see through the command:

[: root] kubelet get sc-n=xxx

Note: only one default storageClass can be set, otherwise the system will cause conflicts because of multiple default and do not know which one to filter as the default, resulting in the default storageClass not taking effect.

5. Dynamic Storage Management practice, GlusterFS

   this case from the definition of storageClass, the creation of storageFS and Heketi services, users apply for PVC to create Pod to use storage resources, storageClass and dynamic resource allocation are described in detail, and the storage mechanism of kubernetes is further analyzed.

   's case here is based on:

Https://www.cnblogs.com/xiaoqshuo/p/10096682.html and the authoritative Guide to kubernetes

(1) Conceptual introduction 1) distributed File system-GlusterFS Overview

   GlusterFS is the core of Scale-Out storage solution Gluster. It is an open source distributed file system with strong scale-out ability to support several PB storage capacity and handle thousands of clients. GlusterFS gathers physically distributed storage resources together with TCP/IP or InfiniBand RDMA networks and uses a single global namespace to manage data. GlusterFS is based on a stackable user space design that provides excellent performance for a wide variety of data loads.

   GlusterFS supports standard clients running standard applications on any standard IP network, and users can access application data using standard protocols such as NFS/CIFS in a globally unified namespace. GlusterFS enables users to get rid of the original independent, high-cost closed storage system, can use ordinary cheap storage devices to deploy storage pools that can be centrally managed, scale out, and virtualized, and the storage capacity can be expanded to the TB/PB level.

   GlusterFS stores data in original data formats (such as EXT3, EXT4, XFS, ZFS) and implements a variety of automatic data repair mechanisms.

Architecture diagram:

Here is just to take you to understand the GlusterFS, easy to understand the content of the PVC mount, there is no need to go too deep.

2) Heketi service

   Heketi service is a framework that provides REST ful API to manage GlusterFS volumes, and can provide dynamic storage resources on cloud platforms such as kubernetes and openstack. It supports GlusterFS multi-cluster management, making it easy for administrators to operate on GlusterFS. The following figure briefly describes the role of Heketi service:

(2) actual combat construction

① installs the GlusterFS client on each node planned for GlusterFS

# yum install glusterfs glusterfs-fuse-y

② enter:-- allow-privileged=true in the startup parameters of the kube-apiserver service of master and the kubelet service on each node of GlusterFS to be started

③ loads specified individual modules

Modprobe dm_snapshot modprobe dm_mirror modprobe dm_thin_pool

④ tags the node where the GlusterFS is to be deployed, in order to deploy the GlusterFS container to the node where GlusterFS is installed

[root@k8s-master01 ~] # kubectl label node k8s-node-1 storagenode=glusterfs node/k8s-node-1 labeled [root@k8s-master01 ~] # kubectl label node k8s-node-2 storagenode=glusterfs node/k8s-node-2 labeled [root@k8s-master01 ~] # kubectl label node k8s-node-3 storagenode=glusterfs node/k8s-node-3 labeled

⑤ creates GlusterFS Management Service Container Cluster

# glusterfs-daemonset.yaml:

Kind: DaemonSetapiVersion: extensions/v1beta1metadata: name: glusterfs labels: glusterfs: daemonset annotations: description: GlusterFS DaemonSet tags: glusterfsspec: template: metadata: name: glusterfs labels: glusterfs-node: pod spec: nodeSelector: storagenode: glusterfs hostNetwork: true containers:-image: gluster/gluster-centos:latest name: glusterfs volumeMounts:-name: glusterfs-heketi MountPath: "/ var/lib/heketi"-name: glusterfs-run mountPath: "/ run"-name: glusterfs-lvm mountPath: "/ run/lvm"-name: glusterfs-etc mountPath: "/ etc/glusterfs"-name: glusterfs-log mountPath: "/ var/log/glusterfs"-name: glusterfs-config mountPath: "/ var/lib/glusterd"-name: glusterfs-dev mountPath: "/ dev"-name: glusterfs-misc mountPath: "/ var/lib/misc/glusterfsd"-name: glusterfs-cgroup mountPath: "/ sys/fs/cgroup" readOnly: true-name: glusterfs-ssl mountPath: "/ etc/ssl" readOnly: true SecurityContext: capabilities: {} privileged: true readinessProbe: timeoutSeconds: 3 initialDelaySeconds: 60 exec: command:-"/ bin/bash"-"- c"-systemctl status glusterd.service livenessProbe: timeoutSeconds: 3 initialDelaySeconds: 60 exec : command:-"/ bin/bash"-"- c"-systemctl status glusterd.service volumes:-name: glusterfs-heketi hostPath: path: "/ var/lib/heketi"-name: glusterfs-run-name: glusterfs-lvm hostPath: path: "/ run/lvm" -name: glusterfs-etc hostPath: path: "/ etc/glusterfs"-name: glusterfs-log hostPath: path: "/ var/log/glusterfs"-name: glusterfs-config hostPath: path: "/ var/lib/glusterd"-name: glusterfs-dev hostPath: path: "/ dev"-name: glusterfs- Misc hostPath: path: "/ var/lib/misc/glusterfsd"-name: glusterfs-cgroup hostPath: path: "/ sys/fs/cgroup"-name: glusterfs-ssl hostPath: path: "/ etc/ssl" # kubelet create-f glusterfs-daemonset.yaml # creation # kubectl get pods # View NAME READY STATUS RESTARTS AGE glusterfs-fvxh7 1 Running 0 47m glusterfs-jjw7b 1 / 1 Running 0 47m glusterfs-td875 1/1 Running 0 47m

⑥ creates a Heketi service

Before deploying Heketi, you need to create a ServiceAccount object for him:

# heketi-service-account.yaml

ApiVersion: v1kind: ServiceAccountmetadata: name: heketi-service-account#kubelet create-f heketi-service-account.yaml

# heketi-deployment-svc.yaml

Kind: DeploymentapiVersion: extensions/v1beta1metadata: name: deploy-heketi labels: glusterfs: heketi-deployment deploy-heketi: heketi-deployment annotations: description: relicas: 1 template: metadata: name: deploy-heketi labels: glusterfs: heketi-pod spec: ServiceAccountName: heketi-service-account containers:-image: heketi/heketi:dev name: deploy-heketi env:- Name: HEKETI_EXECUTOR value: kubernetes-name: HEKETI_FSTAB value: "/ var/lib/heketi/fstab"-name: HEKETI_SNAPSHOT_LIMIT value: "14"-name: HEKETI_KUBE_GLUSTER_DAEONSET value: "y" ports:-containerPort: 8080 volumeMounts:-name: db mountPath: " / var/lib/heketi "readinessProbe: timeoutSeconds: 3 initialDelaySeconds: 3 httpGet: path:" / hello "port: 8080 livenessProbe: timeoutSeconds: 3 initialDelaySeconds: 30 httpGet: path:" / hello "port: 8080 volumes:-name: db hostPath: Path: "/ heketi-data"-kind: ServiceapiVersion: v1metadata: name: deploy-heketi labels: glusterfs: heketi-service deploy-heketi: support annotations: description: Exposes Heketi Service spec: selector: name: deploy-heketi ports:-name: deploy-heketi port: 8080 targetPort: 8080#kubelete create-f heketi-deployment-svc.yaml

⑦ sets up a GlusterFS cluster for Heketi

Before Heketi can manage GlusterFS clusters, you need to set up GlusterFS cluster information for it. You can use the json configuration file to pass in, and Heketi requires a GlusterFS cluster to have at least three nodes.

# topology.json

{"clusters": [{"nodes": [{"node": {"hostnames": {"manage": ["k8s-node-1"] "storage": ["192.168.2.100"]}, "zone": 1} "devices": [{"name": "/ dev/sdb"}]} {"node": {"hostnames": {"manage": ["k8s-node-2"] "storage": ["192.168.2.101"]}, "zone": 1} "devices": [{"name": "/ dev/sdc"}]} {"node": {"hostnames": {"manage": ["k8s-node-3"] "storage": ["192.168.2.102"]}, "zone": 1} "devices": [{"name": "/ dev/sdb"}]}]}

Then go to the container of Heketi and execute:

# export HEKETI_CLI_SERVER= http://localhost:8080#heketi-cli topology load-json= topology.json

After doing this, Heketi completes the creation of the GlusterFS cluster and successfully creates PV and VG on the / dev/ SDB disk on each node of the GlusterFS cluster.

Note: / dev/sdb must be a bare device that does not create a file system.

# View GFS information

[root@k8s-node-1 kubernetes] # heketi-cli topology info (3) Test

After the cluster is successfully created, of course, we should try to see if the function of storageClass can be used:

① defines storageClass

# storageclass-gluster-heketi.yaml

ApiVersion: storage.k8s.io/v1kind: StorageClassmetadata: name: gluster-heketiprovisioner: kubernetes-io/glusterfsparameters: resturl: "http://172.17.2.2:8080" restauthenabled:" false "

② defines PVC

# pvc-gluster-heketi.yaml (using dynamic resource provisioning mode)

Kind: PersistentVolumeClaimapiVersion: v1metadata: name: pvc-gluster-heketispec: storageClassName: gluster-heketi accessModes:-ReadWriteOnce resources: requests: storage: 1Gi

# you can view pv and automatically created PVC through the command

# kubelet get pvc#kubelet get pv

③ mounts PVC using pod

# pod-use-pvc.yaml

ApiVersion: v1kind: Podmetadata: name: pod-use-pvcspec: containers:-name: pod-use-pvc image: busybox command:-sleep-"3600" volumeMounts:-name: gluster-volume mountPath: "/ pv-data" readOnly: false volumes:-name: gluster-volume persistentVolumeClaim: claimName: pvc-gluster-heketi#kubelet create-f pod-use-pvc.yaml # create this pod

Then enter the container:

# kubelet exec-it pod-use-pvc / bin/sh

Create a file in the / pv-data directory and verify that the file is valid in the GlusterFS cluster!

The content of the article is referred to the authoritative Guide to kubernetes.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.