RocketMQ cluster production environment master nodes gracefully offline

1. Description of the status quo

Due to some reasons, a master node in the rocketmq cluster cannot serve properly, so you need to perform traffic migration action for the master node to ensure that messages are not lost. The configuration policy is:

Asynchronous flushing disk

Master-slave asynchronous replication

If the master is taken offline directly, some messages may be lost because of master-slave asynchronous replication, which may cause some messages to be copied to the slave in time. Therefore, the scheme is not feasible.

Another option: turn off the write permission of the broker, and then go offline to the node when the broker no longer has write and consumption.

two。 Turn off broker write permissions

2 indicates write-only permission, 4 indicates read-only permission, and 6 indicates read-write permission

# bin/mqadmin updateBrokerConfig-b broker:port-n nameserver:prot-k brokerPermission-v 4

3. Observe node traffic # bin/mqadmin clusterList-n nameserver:prot

Observe InTPS and OutTPS, the ideal situation is 00:00, and when it no longer changes, the node can be offline.

However, there is no zero in the actual process. InTPS and OutTPS always have values, sometimes single digits and sometimes two digits, most of the time in values of more than 20. At this point, we want to analyze the current consumption state of broker.

4. Observe broker consumption status # sh bin/mqadmin brokerConsumeStats-b broker:prot-n nameserver:prot > > brokerConsumeStats.tmp

Check brokerConsumeStats.tmp, mainly # LastTime and # Diff. Found that the% RETRY% retry class queue # Diff has very small (1 or 3) data, while the other topic is 0. 5%. The latest LastTime time also occurs in the% RETRY% queue. At this point, you can take the node offline.

5.borker read and write permissions restore # bin/mqadmin updateBrokerConfig-b broker:port-n nameserver:prot-k brokerPermission-v 6

Observe whether the traffic of each node is normal.

# bin/mqadmin clusterList-n nameserver:prot