Shulou(Shulou.com)06/02 Report--

This article mainly explains "RocketMQ message loss scenarios and solutions," interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's take you to learn "RocketMQ message loss scenarios and solutions"!

Since MQ is used in the project, it is inevitable to consider message loss. In some situations involving money transactions, message loss can be fatal. So what are the message loss scenarios in RocketMQ?

Let's start with the simplest consumption flow chart:

The above picture roughly contains several scenarios:

Producer generates messages and sends them to RocketMQ. After RocketMQ receives the messages, it must be stored in disk. Otherwise, data loss will occur after power failure or downtime. Consumer obtains messages from RocketMQ. After consumption is successful, the whole process ends.

All three scenarios may result in message loss, as shown in the following figure:

1. When the producer sends a message to Rocket MQ in Scenario 1, if there is network jitter or communication abnormality, the message may be lost.

2. In Scenario 2, the message needs to be persisted to disk. In this case, there are two situations that lead to message loss.

RocketMQ in order to reduce disk IO, will first write messages to os cache, rather than directly to disk, consumers get messages from os cache similar to directly from memory, faster, after a period of time will be asynchronous by os thread will brush messages into disk, at this time is really completed message persistence. In this process, if the message has not completed asynchronous disk flushing, RocketMQ Broker down, it will lead to message loss If the message has been flushed to disk, but the data has not made any backup, once the disk is damaged, then the message will also be lost.

3. The consumer successfully obtains the message from RocketMQ. When the message has not been completely consumed, it notifies RocketMQ that I have consumed the message, and then the consumer is down, but RocketMQ thinks that the consumer has successfully consumed the data, so the data is still lost.

So how do you guarantee zero message loss?

1. In Scenario 1, the solution to ensure that messages are not lost is to use RocketMQ's own transaction mechanism to send messages. The approximate process is as follows:

First, the producer sends a half message to RocketMQ. At this time, the consumer cannot consume the half message. If the half message fails to be sent, the corresponding rollback logic is executed. After the half message is successfully sent, and RocketMQ returns a success response, the producer's core link is executed. If the producer's core link fails to execute, the rollback is executed, and RocketMQ is notified to delete the half message. If the producer's core link succeeds, RocketMQ commits the half message, so that the consumer can consume the data.

There are also some RocketMQ that have not received a response from the producer for a long time to commit/rollback operations, callback the details of the producer interface, interested can refer to:

https://blog.csdn.net/LO_YUN/article/details/101673893

After the producer's message is successfully sent to RocketMQ using RocketMQ transactions, it is guaranteed that messages will not be lost at this stage

2. In Scenario 2, to ensure that messages are not lost, you need to change the asynchronous disk brushing policy of os cache to synchronous disk brushing. In this step, you need to modify the Broker configuration file, change the flushDiskType to SYNC_FLUSH synchronous disk brushing policy, and the default is ASYNC_FLUSH asynchronous disk brushing.

Once synchronous disk brushing returns successfully, it must ensure that the message has been persisted to the disk; in order to ensure that disk damage does not lose data, we need to adopt a master-slave mechanism for RocketMQ, cluster deployment, and backup of data in the Leader in multiple Followers to prevent single point failure.

Search Java Zhiyin public account, reply "back-end interview", send you a Java interview book

3. In Scenario 3, when the message arrives at the consumer, RocketMQ guarantees that the message will not be lost in the code.

//Register Message Listeners Process Messages

consumer.registerMessageListener(new MessageListenerConcurrently() {

@Override

public ConsumeConcurrentlyStatus consumeMessage(List msgs, ConsumeConcurrentlyContext context){

//process messages

return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;

}

});

In the above code, RocketMQ registers a listener in the consumer. When the consumer gets the message, it will call back the listener function to process the message inside.

ConsumeConcurrentlyStatus.CONSUME_SUCCESS will be returned only after your message is processed. Only if CONSUME_SUCCESS is returned, the consumer will tell RocketMQ that I have finished consuming. At this time, if the consumer is down, the message has been processed, and the message will not be lost.

If the consumer fails before CONSUME_SUCCESS is returned, RocketMQ will assume that the consumer node is down and will automatically fail over, passing the message to other consumers in the consumer group to consume the message, ensuring that the message is not lost.

In order to ensure that the message will not be lost, you can directly write the business logic of message consumption in the consummeMessage method. If you have to do some fancy operations, such as the following code

//Register Message Listeners Process Messages

consumer.registerMessageListener(new MessageListenerConcurrently() {

@Override

public ConsumeConcurrentlyStatus consumeMessage(List msgs, ConsumeConcurrentlyContext context){

//Start child threads to process messages asynchronously

new Thread() {

public void run() {

//process messages

}

}.start();

return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;

}

});

If the new child thread processes messages asynchronously, it is possible that the message has not been consumed, and the consumer tells RocketMQ that the message has been consumed, resulting in downtime and loss of the message.

Using the above set of schemes can guarantee zero message loss when using RocketMQ, but performance and throughput will also be greatly reduced.

Using transaction mechanism to transmit messages will take many more steps than ordinary message transmission. Compared with asynchronous disk brushing, one is stored in disk and the other is stored in memory. The speed is not an order of magnitude at all. If the master-slave mechanism is used, the Leader needs to synchronize the data to the Follower. When consuming, it cannot be consumed asynchronously. It can only wait for the consumption to be completed before notifying RocketMQ that the consumption is completed.

Zero message loss is a double-edged sword. It depends on the specific business scenario if you want to use it well. Choosing the right solution is the best.

At this point, I believe that everyone has a deeper understanding of "RocketMQ message loss scenarios and solutions," may wish to actually operate it! Here is the website, more related content can enter the relevant channels for inquiry, pay attention to us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.