Example Analysis of standby_replay High Speed Hot standby status in ceph-mds

This article mainly introduces the ceph-mds standby_replay high-speed hot standby state example analysis, has a certain reference value, interested friends can refer to, I hope you read this article after a great harvest, the following let Xiaobian take you to understand.

MDS for ceph is a metadata service for the cephFS file storage service.

When cephfs is created, there is a ceph-mds service to manage it. By default ceph assigns an mds service to manage cephfs, even if multiple mds services have been created, as follows:

[root@ceph-admin my-cluster]# ceph-deploy mds create ceph-node01 ceph-node02....... [root@ceph-admin ~]# ceph -s cluster: id: 06dc2b9b-0132-44d2-8a1c-c53d765dca5d health: HEALTH_OK services: mon: 2 daemons, quorum ceph-admin,ceph-node01 mgr: ceph-admin(active) mds: mytest-fs-1/1/1 up {0=ceph-admin=up:active}, 2 up:standby osd: 3 osds: 3 up, 3 in rgw: 2 daemons active data: pools: 8 pools, 64 pgs objects: 299 objects, 137 MiB usage: 3.4 GiB used, 297 GiB / 300 GiB avail pgs: 64 active+clean

At this point, only ceph-admin in mds is active, and the rest are standby.

Standby is almost a disaster recovery state, but in fact the switching speed is relatively slow, which will definitely affect the actual business system.

The tests were as follows:

[root@ceph-admin my-cluster]# killall ceph-mds[root@ceph-admin my-cluster]# ceph -s cluster: id: 06dc2b9b-0132-44d2-8a1c-c53d765dca5d health: HEALTH_WARN 1 filesystem is degraded services: mon: 2 daemons, quorum ceph-admin,ceph-node01 mgr: ceph-admin(active) mds: mytest-fs-1/1/1 up {0=ceph-node02=up:rejoin}, 1 up:standby osd: 3 osds: 3 up, 3 in rgw: 2 daemons active data: pools: 8 pools, 64 pgs objects: 299 objects, 137 MiB usage: 3.4 GiB used, 297 GiB / 300 GiB avail pgs: 64 active+clean [root@ceph-admin my-cluster]# ceph -s cluster: id: 06dc2b9b-0132-44d2-8a1c-c53d765dca5d health: HEALTH_OK services: mon: 2 daemons, quorum ceph-admin,ceph-node01 mgr: ceph-admin(active) mds: mytest-fs-1/1/1 up {0=ceph-node02=up:active}, 1 up:standby osd: 3 osds: 3 up, 3 in rgw: 2 daemons active data: pools: 8 pools, 64 pgs objects: 299 objects, 137 MiB usage: 3.4 GiB used, 297 GiB / 300 GiB avail pgs: 64 active+clean io: client: 20 KiB/s rd, 3 op/s rd, 0 op/s wr

After the active mds are killed, the standby mds become active after the rejoin state, about 3- 5 seconds later. In a production environment, metadata tends to be longer due to the larger amount of data.

To make mds switch faster, we need to switch our mds service to standby_replay state. The official explanation for this state is as follows:

The MDS is following the journal of another up:active MDS. Should the active MDS fail, having a standby MDS in replay mode is desirable as the MDS is replaying the live journal and will more quickly takeover. A downside to having standby replay MDSs is that they are not available to takeover for any other MDS that fails, only the MDS they follow.

In fact, standby_replay will update synchronously according to the active mds metadata log in real time, which can speed up the switching rate, so how to make mds run in standby_replay state?

[root@ceph-node01 ~]# ps aufx|grep mdsroot 700547 0.0 0.0 112704 976 pts/1 S+ 13:45 0:00 \_ grep --color=auto mdsceph 690340 0.0 0.5 451944 22988 ? Ssl 10:09 0:03 /usr/bin/ceph-mds -f --cluster ceph --id ceph-node01 --setuser ceph --setgroup ceph[root@ceph-node01 ~]# killall ceph-mds[root@ceph-node01 ~]# /usr/bin/ceph-mds -f --cluster ceph --id ceph-node01 --setuser ceph --setgroup ceph --hot-standby 0starting mds.ceph-node01 at -

We manually shut down the ceph-mds service and restarted the mds service by adding the--hot-standby parameter.

Next, you can see that the mds of ceph-node02 is already in standby-replay state:

[root@ceph-admin my-cluster]# ceph -s cluster: id: 06dc2b9b-0132-44d2-8a1c-c53d765dca5d health: HEALTH_OK services: mon: 2 daemons, quorum ceph-admin,ceph-node01 mgr: ceph-admin(active) mds: mytest-fs-1/1/1 up {0=ceph-admin=up:active}, 1 up:standby-replay, 1 up:standby osd: 3 osds: 3 up, 3 in rgw: 2 daemons active data: pools: 8 pools, 64 pgs objects: 299 objects, 137 MiB usage: 3.4 GiB used, 297 GiB / 300 GiB avail pgs: 64 active+clean io: client: 851 B/s rd, 1 op/s rd, 0 op/s wr

Or, of course, you can use the ceph-mds startup script to automatically put the service in a standby-replay state when it starts

Thank you for reading this article carefully. I hope that Xiaobian's "sample analysis of standby_replay high-speed hot standby state in ceph-mds" will be helpful to everyone. At the same time, I hope that everyone will support you a lot and pay attention to the industry information channel. More relevant knowledge is waiting for you to learn!