Shulou(Shulou.com)06/03 Report--

Reference article:

Https://ieevee.com/tech/2018/05/16/k8s-rbd.html

Https://zhangchenchen.github.io/2017/11/17/kubernetes-integrate-with-ceph/

Https://docs.openshift.com/container-platform/3.5/install_config/storage_examples/ceph_rbd_dynamic_example.html

Https://jimmysong.io/kubernetes-handbook/practice/using-ceph-for-persistent-storage.html

Thanks to the technical reference provided by the above author, I sort it out here and realize the implementation of multi-master database cluster and master-slave database combined with Ceph RDB respectively. The following configuration is for testing only and cannot be used as a production configuration.

Classification of storage in K8S

There are mainly the following categories in the persistent storage of K8S:

Volume: components mounted directly on pod. All other storage components in K8s are directly linked to pod through volume. Volume has a type attribute, and type determines what kind of storage is mounted, such as emptyDir,hostPath,nfs,rbd, persistentVolumeClaim, and so on. Unlike the concept of volume in docker, the life cycle of volume in docker is tightly tied to docker. Here, the life cycle varies according to the type. For example, if the emptyDir type is the same as the docker, the pod is dead and the corresponding volume is gone, while other types are permanent storage. For more information, please refer to Volumes.

Persistent Volumes: as the name implies, this component is used to support persistent storage, and the Persistent Volumes component abstracts the provider of the back-end storage (that is, the type in the volume above) and the consumer (that is, which pod is used). This component provides the concepts of PersistentVolume and PersistentVolumeClaim to abstract the two mentioned above. A PersistentVolume (referred to as PV) is a storage space provided by backend storage. In ceph rbd, it is an image. A PersistentVolumeClaim (PVC for short) can be regarded as a user's request for PV. The PVC will be bound to a PV, and then a specific pod will mount the PVC in the volume, and the corresponding PV will be mounted.

Dynamic Volume Provisioning: dynamic volume discovery, such as Persistent Volumes above, we must first create a storage block, such as an image in ceph, and then bind the image to PV before we can use it. This static binding mode is so rigid that each request for storage requires a storage fast from the storage provider. Dynamic Volume Provisioning is the solution to this problem. It introduces the concept of StorageClass, StorageClass abstracts the storage provider, just specify the StorageClass in the PVC, and then specify how much storage you want, and the storage provider will dynamically create the required storage fast according to the requirements. Even, we can specify a default StorageClass so that we only need to create a PVC. The configuration initialization environment already has a k8s cluster and already has a Ceph cluster where all nodes install ceph-common

Add the yum source for ceph:

[Ceph] name=Ceph packages for $basearchbaseurl= https://mirrors.aliyun.com/ceph/rpm-mimic/el7/$basearchenabled=1gpgcheck=1type=rpm-mdgpgkey=https://download.ceph.com/keys/release.asc[Ceph-noarch]name=Ceph noarch packagesbaseurl= https://mirrors.aliyun.com/ceph/rpm-mimic/el7/noarchenabled=1gpgcheck=1type=rpm-mdgpgkey=https://download.ceph.com/keys/release.asc[ceph-source]name=Ceph source packagesbaseurl= https://mirrors.aliyun.com/ceph/rpm-mimic/el7/SRPMSenabled=1gpgcheck=1type=rpm-mdgpgkey=https://download.ceph.com/keys/release.asc

Install ceph-common:

Yum install ceph-common-y

If a dependency error occurs during the installation process, it can be resolved in the following ways:

Yum install-y yum-utils & &\ yum-config-manager-- add-repo https://dl.fedoraproject.org/pub/epel/7/x86_64/ & &\ yum install-- nogpgcheck-y epel-release & &\ rpm--import / etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 &\ rm-f / etc/yum.repos.d/dl.fedoraproject.org*yum-y install ceph-common configuration ceph configuration file

Copy the ceph configuration file to the node node of each k8s

[root@ceph-1 ~] # scp / etc/ceph k8s-node:/etc/ Test volume

By using a simple volume to test whether the cluster environment is normal or not, in practical applications, the data that needs to be permanently saved cannot use volume.

Create an images in a Ceph cluster

When creating a new mirror, you need to disable some unsupported attributes:

Rbd create foobar-s 1024-p k8s rbd feature disable k8s/foobar object-map fast-diff deep-flatten

View image information:

# rbd info k8s/foobarrbd image 'foobar': size 1 GiB in 256 objects order 22 (4 MiB objects) id: ad9b6b8b4567 block_name_prefix: rbd_data.ad9b6b8b4567 format: 2 features: layering, exclusive-lock op_features: flags: create_timestamp: Tue Apr 23 17:37:39 2019 mount volume directly using POD

The admin.keyring file of ceph is specified as the authentication key:

# cat test.yaml apiVersion: v1kind: Podmetadata: name: rbdspec: containers:-image: nginx name: rbd-rw volumeMounts:-name: rbdpd mountPath: / mnt volumes:-name: rbdpd rbd: monitors:-'192.168.20.41VR 6789' pool: K8s image: foobar fsType: xfs readOnly: false user: admin Keyring: / etc/ceph/ceph.client.admin.keyring uses PV and PVC

If you need to save the data permanently (it will not be lost when the pod is deleted), we need to use PV (PersistentVolume) and PVC (PersistentVolumeClaim).

Create imagesrbd create-s 1024 k8s/pvrbd feature disable k8s/pv object-map fast-diff deep-flatten in a Ceph cluster

View image information:

# rbd info k8s/pvrbd image 'pv': size 1 GiB in 256 objects order 22 (4 MiB objects) id: adaa6b8b4567 block_name_prefix: rbd_data.adaa6b8b4567 format: 2 features: layering, exclusive-lock op_features: flags: create_timestamp: Tue Apr 23 19:09:58 2019 create a secret to generate an encrypted keygrep key / etc/ceph/ceph.client.admin.keyring | awk' {printf "% s" $NF}'| base64 will generate key to create a SecretapiVersion: v1kind: Secretmetadata: name: ceph-secrettype: "kubernetes.io/rbd" data: key: QVFBbk1MaGNBV2laSGhBQUVOQThRWGZyQ3haRkJDNlJaWTNJY1E9PQ==--- create PV and PVC files # cat ceph-rbd-pv.yaml apiVersion: v1kind: PersistentVolumemetadata: name: ceph-rbd-pvspec: capacity: storage: 1Gi accessModes:-ReadWriteOnce rbd: monitors:-'192.168.20.41 67 key' pool: K8s image: pv User: admin secretRef: name: ceph-secret fsType: xfs readOnly: false persistentVolumeReclaimPolicy: Recycle# cat ceph-rbd-pvc.yaml apiVersion: v1kind: PersistentVolumeClaimmetadata: name: ceph-rbd-pv-claimspec: accessModes:-ReadWriteOnce resources: requests: storage: 1Gi create POD# cat test3-pvc.yaml apiVersion: v1kind: Podmetadata: name: rbd-nginxspec: containers:-image: nginx name: rbd-rw volumeMounts:-name: Rbd-pvc mountPath: / mnt volumes:-name: rbd-pvc persistentVolumeClaim: claimName: ceph-rbd-pv-claim the role of using StorageClassStorage Class

To put it simply, storage is configured to access IP/Port, user name, keyring, pool, and other information of ceph RBD. We do not need to create image; in advance. When a user creates a PVC, K8s looks for whether there is a storage class type that meets the PVC request. If so, do the following in turn:

Create an image on the ceph cluster to create a PV named pvc-xx-xxx-xxx, the size of the storage of the pvc request. Bind the above PV to PVC, format it and hang it in the container

In this way, the administrator only needs to create the storage class, and the user can take care of the rest by himself. If you want to prevent resources from being exhausted, you can set up Resource Quota.

When pod needs a volume, it can be declared directly through PVC, and persistent volumes can be created according to the requirements.

Create storageclass # cat storageclass.yaml apiVersion: storage.k8s.io/v1kind: StorageClassmetadata: name: fastprovisioner: kubernetes.io/rbdparameters: monitors: 192.168.20.41 userId 6789 adminId: admin adminSecretName: ceph-secret pool: K8s userId: admin userSecretName: ceph-secret fsType: xfs imageFormat: "2" imageFeatures: "layering" create PVC

RBD only supports ReadWriteOnce and ReadOnlyAll, not ReadWriteAll. Note that the difference between the two is whether different nodes can be mounted simultaneously. On the same node, even ReadWriteOnce can be mounted to two containers at the same time.

When you create an application, you need to create both pv and pod, both of which are associated through storageClassName. In pvc, you need to specify that its storageClassName is the name (that is, fast) of the sc created above.

# cat pvc.yaml kind: PersistentVolumeClaimapiVersion: v1metadata: name: rbd-pvc-pod-pvcspec: accessModes:-ReadWriteOnce volumeMode: Filesystem resources: requests: storage: 1Gi storageClassName: fast

Create pod

# cat pod.yaml apiVersion: v1kind: Podmetadata: labels: test: rbd-pvc-pod name: ceph-rbd-sc-pod1spec: containers:-name: ceph-rbd-sc-nginx image: nginx volumeMounts:-name: ceph-rbd-vol1 mountPath: / mnt readOnly: false volumes:-name: ceph-rbd-vol1 persistentVolumeClaim: claimName: rbd-pvc-pod-pvc

When using Storage Class, in addition to using PVC to declare the persistent volume to use, you can also declare creation by creating a volumeClaimTemplates (storage settings in StatefulSets), and if multiple copies are involved, you can use StatefulSets configuration:

ApiVersion: apps/v1kind: StatefulSetmetadata: name: nginxspec: selector: matchLabels: app: nginx serviceName: "nginx" replicas: 3 template: metadata: labels: app: nginxspec: terminationGracePeriodSeconds: 10 containers:-name: nginx image: nginx volumeMounts:-name: www mountPath: / usr/share/nginx/html volumeClaimTemplates:-metadata: name: www Spec: accessModes: ["ReadWriteOnce"] storageClassName: "fast" resources: requests: storage: 1Gi

But be careful not to use Deployment. Because, if the number of replicas of Deployment is 1, then it can still be used, which is the same as Pod; but if the number of replicas is more than 1, you will find that only one Pod is started and all other Pod is in ContainerCreating state after creating deployment. After a while, describe pod can see it, but he hasn't been waiting for volume for a long time.

Example 1: create a mysql-galera cluster (multi-host)

Official document: https://kubernetes.io/docs/tasks/run-application/run-replicated-stateful-application/

Introduction to statefulset

Statefulset (called petset before 1. 5), statefulset and deployment,replicasets are at the same level. However, Deployments and ReplicaSets are designed for stateless services. Statefulset is to solve the problem of stateful service. Its application scenarios are as follows:

Stable persistent storage, that is, Pod can still access the same persistent data after rescheduling. It is based on PVC to achieve a stable network flag, that is, its PodName and HostName remain unchanged after Pod rescheduling, based on Headless Service (that is, Service without Cluster IP). Orderly deployment, orderly expansion, that is, Pod is sequentially deployed or expanded according to the defined order (that is, from 0 to Nmuri 1, before the next Pod runs-all previous Pod must be Running and Ready state), based on init containers. Orderly contraction, orderly deletion (that is, from Nmuri 1 to 0).

From the application scenario, statefuleset is especially suitable for database clusters such as mqsql,redis. Accordingly, a statefuleset has the following three parts:

The HeadlessService used to define the network logo (DNS domain), refer to the documentation) the volumeClaimTemplates used to create the PersistentVolumes defines the application-specific StatefulSet1. Generate and create ceph secret

You can skip this step if the secret of ceph has been created in the k8s cluster

Generate an encrypted key

Grep key / etc/ceph/ceph.client.admin.keyring | awk'{printf "% s", $NF}'| base64

Create a Secret from the generated key

ApiVersion: v1kind: Secretmetadata: name: ceph-secret namespace: galeratype: "kubernetes.io/rbd" data: key: QVFBbk1MaGNBV2laSGhBQUVOQThRWGZyQ3haRkJDNlJaWTNJY1E9PQ==---2. Create StorageClass# cat storageclass.yaml apiVersion: storage.k8s.io/v1kind: StorageClassmetadata: name: fastprovisioner: kubernetes.io/rbdparameters: monitors: 192.168.20.41 admin userSecretName: admin adminSecretName: admin adminSecretName: K8s userId: admin userSecretName: xfs imageFormat: "2" imageFeatures: "layering" 3. Create headless Service

Galera-service.yaml

ApiVersion: v1kind: Servicemetadata: annotations: service.alpha.kubernetes.io/tolerate-unready-endpoints: "true" name: galera namespace: galera labels: mysqlspec: ports:-port: 3306 name: mysql # *. Galear.default.svc.cluster.local clusterIP: None selector: app: mysql4. Create statefulset

The V1 version of StatefulSet is used here. Compared with the previous version, the v1 version is the current stable version. The difference from the previous beta version is that the v1 version needs to add the parameter of spec.selector.matchLabels, which needs to be consistent with spec.template.metadata.labels.

ApiVersion: apps/v1kind: StatefulSetmetadata: name: mysql namespace: galeraspec: selector: matchLabels: app: mysql serviceName: "galera" replicas: 3 template: metadata: labels: app: mysql spec: initContainers:-name: install image: mirrorgooglecontainers/galera-install:0.1 imagePullPolicy: Always args:-"- work-dir=/work-dir" volumeMounts: -name: workdir mountPath: "/ work-dir"-name: config mountPath: "/ etc/mysql"-name: bootstrap image: debian:jessie command:-"/ work-dir/peer-finder" args:-- on-start= "/ work-dir/on-start.sh"-"- service=galera" Env:-name: POD_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace volumeMounts:-name: workdir mountPath: "/ work-dir"-name: config mountPath: "/ etc/mysql" containers:-name: mysql image: mirrorgooglecontainers/mysql-galera:e2e Ports:-containerPort: 3306 name: mysql-containerPort: 4444 name: sst-containerPort: 4567 name: replication-containerPort: 4568 name: ist args:-- defaults-file=/etc/mysql/my-galera.cnf-user=root readinessProbe: # TODO: If docker exec is buggy just use gcr. Io/google_containers/mysql-healthz:1.0 exec: command:-sh-- c-"mysql- u root-e'show databases '"initialDelaySeconds: 15 timeoutSeconds: 5 successThreshold: 2 volumeMounts:-name: datadir mountPath: / var/lib/-name: config mountPath: / etc/mysql volumes:-name: config emptyDir: {}-name: workdir emptyDir: {} volumeClaimTemplates:-metadata: name: datadir annotations: Volume.beta.kubernetes.io/storage-class: "fast" spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 1Gi5. Check pod

Check that the pod status is normal

[root@master-1] # kubectl get pod-n galera NAME READY STATUS RESTARTS AGEmysql-0 1 + 1 Running 0 48mmysql-1 1 + + 1 Running 0 43mmysql-2 1 + + 1 Running 0 38m

Database cluster establishment:

[root@master-1] # kubectl exec mysql-1-n galera-- mysql- uroot-e'show status like "wsrep_cluster_size"; 'Variable_name Valuewsrep_cluster_size 3

View pv bindings:

[root@master-1 mysql-cluster] # kubectl get pvc- l app=mysql-n galeraNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGEdatadir-mysql-0 Bound pvc-6e5a1c45-666b-11e9-ad20-000c29016590 1Gi RWO fast 3d20hdatadir-mysql-1 Bound pvc-25683cfd-666c-11e9-ad20-000c29016590 1Gi RWO Fast 3d20hdatadir-mysql-2 Bound pvc-c024b422-666c-11e9-ad20-000c29016590 1Gi RWO fast 3d20h

Test database:

Kubectl exec mysql-2-n galera-- mysql- uroot-e change_master_to.sql.in fi # Check if we need to complete a clone by starting replication. If [[- f change_master_to.sql.in]]; then echo "Waiting for mysqld to be ready (accepting connections)" until mysql-h 127.0.0.1-e "SELECT 1"; do sleep 1; done echo "Initializing replication from clone position" # In case of container restart, attempt this at-most-once. Mv change_master_to.sql.in change_master_to.sql.orig mysql-h 127.0.0.1

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.