Shulou(Shulou.com)06/01 Report--

How to achieve Elasticsearch data persistence in cephfs? in view of this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

1. Cephfs

Cephfs creation

A three-node ceph cluster is built on 172.20.0.10.

10.0.169.87 node1.cephfs-cluster (mons at {node1=10.0.169.87:6789/0})

10.0.149.141 node2.cephfs-cluster

10.0.235.158 node3.cephfs-cluster

A 300G data cloud disk (using lvm to create partitions to facilitate later expansion) is mounted on each node, and the formatted file system is mounted to / var/local/osd [123] on each node.

[root@node1 ~] # mount-l | grep cephfs

/ dev/mapper/vg--for--cephfs-lvm--for--cephfs on / var/local/osd1 type xfs (rw,relatime,seclabel,attr2,inode64,noquota)

[root@node2 ~] # mount-l | grep cephfs

/ dev/mapper/vg--for--cephfs-lvm--for--cephfs on / var/local/osd2 type xfs (rw,relatime,seclabel,attr2,inode64,noquota)

[root@node3 ~] # mount-l | grep cephfs

/ dev/mapper/vg--for--cephfs-lvm--for--cephfs on / var/local/osd3 type xfs (rw,relatime,seclabel,attr2,inode64,noquota)

Later, you can refer to the two articles given earlier to complete the creation of the cephfs, and then mount the cephfs locally.

[root@node1] # mount-t ceph: 6789 opt-o name=admin,secret=

Where admin-key is obtained in the following way

[root@node1 opt] # ceph auth get client.admin

[root@node1 ~] # mount-l | grep opt

10.0.169.87 on / opt type ceph (rw,relatime,name=admin,secret=,acl,wsize=16777216)

Then create the next three directories in cephfs

[root@node1 opt] # pwd

/ opt

[root@node1 opt] # ll

Total dosage 0

Drwxr-xr-x 1 1000 root 1 August 1 22:07 elasticsearch_node1

Drwxr-xr-x 1 1000 root 1 August 1 22:07 elasticsearch_node2

Drwxr-xr-x 1 1000 root 1 August 1 22:07 elasticsearch_node3

II. Persistent storage

It is easy to use and is recommended.

There are the following files in the elasticsearch/persistent_storage directory.

Drwxr-xr-x. 2 root root 30 July 31 14:42 cephfs (include file-rw-r--r--. 1 root root 177.30 15:20 ceph-secret.yaml)

-rw-r--r--. 1 root root 1115 July 31 15:50 replicset_elasticsearch_1.yaml

-rw-r--r--. 1 root root 1115 July 31 15:50 replicset_elasticsearch_2.yaml

-rw-r--r--. 1 root root 1115 July 31 15:50 replicset_elasticsearch_3.yaml

The secret file of cephfs is used to create secret resources in K8s and save the information such as key that accesses cephfs.

You need to pay attention to the following in replicaset_elasticsearch_1.yaml

VolumeMounts:

-name: cephfs

MountPath: / usr/share/elasticsearch/data # Mount the elsticsearch_node1 in cephfs to the / usr/share/elasticsearch/data directory in pod elasticsearch_node1.

Volumes:

-name: cephfs

Cephfs:

Monitors:

-10.0.169.87 cephfs management node

Path: / elasticsearch_node1 # directory created earlier

SecretRef: # reference the secret that accesses cephfs

Name: ceph-secret

2.persistent storage architecture diagram

Data data is permanently saved to cephfs unless the relevant index; is deleted in elasticsearch or the cephfs is mounted locally, and then the contents of elasticsearch_node [123] are deleted.

Special attention should be paid to the permissions of the elasticsearch_node [123] directory. The directory user created in the previous way is root:root, but in elasticsearch, the elasticsearch user is required to run the elasticsearch process and save the data in data. In the docker image of elasticsearch, the operation to change the directory owner has been added, changing the owner of elasticsearch_node [123] to elasticsearch:root, and the owner seen by external mapping is 1000:root.

3. Why create 3 replicaset to manage 3 elasticsearch pod instances instead of setting replicas to 3 in one replicaset?

Before adding cephfs storage, one replicaset was used to manage three elasticsearch pod.

Each elasticsearch needs different space to store its own data data, and replicas cannot be set to 3 in replicaset, which will cause the / usr/share/elasticsearch/data directory of each elasticsearch to be mounted to the same cephfs directory. It is not recommended to create three container directories in one pod to mount different cephfs. This will work, but these container on the same worker node will cause the node to be overloaded.

III. Persistent volume and persistent volume claim

The principle of pv/pvc is similar to that of persistent storage, only the relevant principles are given here for easy understanding.

Pv/pvc architecture diagram

The following points are required

Like pv/pvc, persitent storage uses the posix API provided by cephfs to mount the cephfs to the corresponding worker node, and then mounts the directory to the container using docker volume. You can use docker inspect to view the mount information of the container on the corresponding worker node.

The size declared in pv/pvc may not be the size in cephfs (usually not), and how much data can be stored depends on the cephfs. The storage size declared by pv is for pvc to use when selecting an pv that meets its needs.

Pv is a type of cluster level-level resource that does not belong to any namespace, so it can be used by pvc in any namespace.

It has been verified that the persistentVolumeReclaimPolicy of pv with cephfs as backend storage only supports Retain, but not Delete and Recycle. This means that after deleting the pvc, the data in the pv (actually the data stored in the cephfs file system) will not be deleted, but at this point the pv can no longer be used by any pvc, and the pv can only be deleted and recreated. Because the data is stored in the cephfs file system, you don't have to worry about losing it. Delete deletes the data, and Recycle deletes the data and recreates the pv to provide services to other pvc.

IV. Later stage optimization

Cephfs read and write performance.

The read and write performance of cephfs needs to be optimized, otherwise elasticsearch will take a long time to initialize. If it is found that elasticsearch cannot be initialized all the time (kubectl get pod finds that the number of ready is 0), the initialDelaySeconds detected by liveness may be too short, so that the elasticsearch is dropped by liveness kill before initialization is completed and the relevant pod is restarted. You can set this value to a longer value.

This is the answer to the question about how to achieve Elasticsearch data persistence in cephfs. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel to learn more about it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.