Shulou(Shulou.com)06/01 Report--

This article will explain in detail about the repeated consumption of messages in MQ and how to solve them. The content of the article is of high quality, so the editor will share it with you for reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

Reasons for repeated consumption

Because in the case of network delay, the problem of repeated message transmission is inevitable, if you have to realize that the message can not be sent repeatedly, it is basically too difficult, because the network environment is unpredictable and will increase the complexity of the program. therefore, messages are allowed to be sent repeatedly by default.

As long as data is exchanged over the network, this problem cannot be avoided. So the way to solve this problem is to bypass it. Then the question becomes: what should be done if the consumer receives two identical messages?

Examples of repeated consumption problems

The problem of repeated consumption of messages may occur in RabbitMQ, RocketMQ and Kafka, which is normal. Because this problem is not usually guaranteed by MQ itself, it is guaranteed by our development.

RabbitMQ

RabbitMQ does not guarantee that messages will not be duplicated. If your business needs to guarantee strict non-repetition of messages, you need to repeat them on the business side.

AMQP consumer confirmation mechanism

AMQP defines a consumer confirmation mechanism (message ack). If a consumer application crashes (at this point the connection will be broken, broker will know), but broker has not yet obtained the ack, then the message will be put back into the queue. Therefore, AMQP provides "at least one delivery" (at-least-once delivery). In an exception, the message will be consumed repeatedly, and the business needs to be idempotent (duplicate message processing).

Message repeated release: does not exist because AMQP defines a tx transaction to ensure that production messages are received by broker and successfully queued. The TX transaction is a blocking call, and the producer has to wait for the confirmation that broker writes to the disk before continuing to send the message. When a transaction fails (such as a broker downtime scenario), broker does not guarantee that all submitted messages will be queued. RabbitMQ uses the confirm mechanism to optimize the acknowledgement of production messages (messages can be posted continuously, but batch acknowledgements can be replied).

Message repeat consumption: AMQP provides "at least one delivery" (at-least-once delivery). In abnormal cases, messages will be consumed repeatedly, and the business needs to be idempotent (duplicate message processing).

Kafka

Kafka actually has the concept of offset, that is, each message is written with an offset, which represents the serial number of the message. Then after consumer consumes the data, it will submit the offset of the consumed message at regular intervals (periodically).

Suppose there is such a scene. When the data enters the kafka,kafka on 1-2-3, each of the three pieces of data will be assigned an offset, which represents the serial number of the data, and the assigned offset is 152153154 in turn. When consumers go to spend from kafka, they also spend in this order. If the consumer consumes this piece of data from offset=153 and is just about to submit the offset to zookeeper, the consumer process is restarted. Then the data consumed at this time 1 offset 2 has not been submitted, kafka will not know that you have consumed this data offset=153. After the restart, consumers will look for kafka to continue to pass on the data behind the place where they spent last time. Data 1 # 2 is consumed again.

If what consumers do is to take a piece of data and write one into the database, it will lead to saying that you may insert data 1 and 2 into the database twice, then the data is wrong. In fact, repeated consumption is not terrible, the terrible thing is that you do not consider the repeated consumption, how to ensure idempotency.

Solution

The business logic of the consumer side to process the message remains idempotent.

Idempotent, to put it colloquially, just one data, or a request, is repeated for you many times, and you have to make sure that the corresponding data is not changed and cannot go wrong.

For example, you get this message to do insert operation of the database. That's easy. Make a unique primary key for this message, and even if there is repeated consumption, it will lead to primary key conflicts and avoid dirty data in the database.

For example, if you get this message to do the set operation of redis, it will be easy and need not be solved, because no matter how many times you set, the result is the same, set operation is idempotent.

If the above two situations don't work, make a big move. Prepare a third-party media to record consumption. Take redis as an example, a global id is assigned to the message, and as long as the message has been consumed, it will be written to redis in the form of Kmurv. Before consumers start spending, they can go to redis to check whether there is a record of consumption.

So much for the sharing of repeated consumption and solution of messages in MQ. I hope the above content can be helpful to you and learn more. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.