Shulou(Shulou.com)06/01 Report--

This article mainly introduces how to delete and add osd nodes in ceph, which has a certain reference value, and interested friends can refer to it. I hope you will gain a lot after reading this article.

In ceph, data is stored in osd in terms of PG. In general, an osd corresponds to a disk. When the osd node dies, if the osd data disk is not damaged, the osd data disk can be used to recover the data.

Now the situation is as follows: disable the osd.8 and osd.9 of the osd2 node, which means that the osd2 node is dead.

The health status of the cluster is as follows:

1 delete osd.8,osd.9 from crush, and delete the corresponding osd, and auth,host

Ceph osd crush rm osd.8

Ceph osd crush rm osd.9

Ceph osd rm 8

Ceph osd rm 9

Ceph auth del osd.8

Ceph auth del osd.9

Ceph osd crush rm osd2

After removing the failed node and osd from the cluster, the new cluster is shown in the following figure

2 create a new osd2 node

Mount the data disk to the new osd2 node and rebuild the osd

2.1 Mount the data disk to a temporary directory

Mount the data disk to the / var/lib/ceph/osd/tmp/mnt directory. If the directory does not exist, create a new one.

Mkdir / var/lib/ceph/tmp/mnt

Mount / dev/sdb1 / var/lib/ceph/tmp/mnt

Read the fsid,whoami file in the / var/lib/ceph/tmp/mnt directory, that is, the uuid and osd-num of the osd corresponding to the disk

Cat fsid whoami

F187533d-d3cd-4e40-8416-a93c9080876c

eight

2.2 rebuilding osd

After reading fsid and osd-num, you can use this information to rebuild osd. You need to read the / etc/ceph.conf file to ensure that the file exists and is consistent with the file in the cluster.

Ceph osd create f187533d-d3cd-4e40-8416-a93c9080876c 8

If the error appears as shown in the figure, the keyring file was not found, which is the authentication file between clusters.

2017-10-16 16 unable to find a keyring on 47purl 15.681089 7f84f1a80700-1 auth: unable to find a keyring on / etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory

2017-10-16 16 7f84f1a80700 47charge 15.681315 7f84f1a80700-1 monclient (hunting): ERROR: missing keyring, cannot use cephx for authentication

2017-10-16 16 Flux 47 client.admin initialization error 15.681404 7f84f1a80700 0 librados: client.admin initialization error (2) No such file or directory

Error connecting to cluster: ObjectNotFound

You can copy a copy of this file from other nodes in the cluster (under the / etc/ceph directory of the primary node) and put it in the / etc/ceph directory

Scp ceph.client.admin.keyring osd2:/etc/ceph

Create osd again

Ceph osd create f187533d-d3cd-4e40-8416-a93c9080876c 8

Return 8 indicates that the creation is successful.

Create a new osd mount point

Mkdir / var/lib/ceph/osd/ceph-8

Uninstall temporary mount directory

Umount / var/lib/ceph/tmp/mnt

Mount the data disk to the osd directory

Mount / dev/sdb1 / var/lib/ceph/osd/ceph-8

Register the key for this OSD

Ceph auth add osd.8 osd 'allow *' mon 'allow rwx'-I / var/lib/ceph/osd/ceph-8/keyring

Added key for osd.8

3 join the cluster

Add new nodes to crushmap

Ceph osd crush add-bucket osd2 host

Place the pin under the default root

Ceph osd crush move osd2 root=default

Osd joins crushmap

Ceph osd crush add osd.8 0.00980 host=osd2

Here, the weight given to osd.8 is 0.00980, which is known to other osd in the cluster. Inconsistent weights may cause the state of pg to remain in unclean all the time.

Now that osd is in the cluster, the osd that joins the cluster can receive data

Root@mon1:/etc/ceph# ceph osd tree

ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY

-1 0.05878 root default

-2 0.01959 host osd1

1 0.00980 osd.1 up 1.00000 1.00000

3 0.00980 osd.3 up 1.00000 1.00000

-5 0.01959 host osd3

4 0.00980 osd.4 up 1.00000 1.00000

5 0.00980 osd.5 up 1.00000 1.00000

-3 0.01959 host osd2

9 0.00980 osd.9 up 1.00000 1.00000

8 0.00980 osd.8 down 0 1.00000

Osd.8 is still in down state because the process of osd.8 has not been started yet

Start the process

Systemctl start ceph-osd@8

Now the health status and osd distribution of the cluster, as shown in the following figure

As shown in the figure above, the osd node is restored successfully.

Thank you for reading this article carefully. I hope the article "how to delete and add osd nodes in ceph" shared by the editor will be helpful to you. At the same time, I also hope you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.