Shulou(Shulou.com)06/03 Report--

This article mainly introduces "how to use the version number to ensure the idempotency of MQ consumption messages". In daily operation, I believe that many people have doubts about how to use version numbers to ensure the idempotency of MQ consumption messages. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the question of "how to use the version number to ensure the idempotency of MQ consumption messages"! Next, please follow the editor to study!

Soul torture

Why does the consumption of MQ messages sometimes require idempotency?

You all say that version numbers can be used to solve idempotent consumption?

What is the fundamental problem of idempotent consumption of information?

With the increasing complexity of the system, most systems will introduce MQ to decouple. In fact, from the original intention of introducing MQ, most systems are to decouple the complexity brought by multiple modules, while some "architects" say: to solve performance problems. Of course, I do not rule out that MQ has the role of traffic peaking, I am just saying that the original intention of most systems to introduce MQ is system decoupling.

When a large single system is gradually divided into multiple small systems, that is, the so-called micro-service split, the communication between micro-services and distributed transactions almost all need the support of MQ, which fully reflects the importance of MQ in distributed systems. At this time, the interaction between the whole system is similar to that shown in the following figure.

Image

Production message

Now that the MQ component is introduced, it must mean that there are both producers and consumers of messages, which is also a typical subscription model. Throughout the life cycle of the message data, it passes through the producer = "MQ=" consumer, in turn, three main parts. From the producer's point of view, reliable delivery of messages is the top priority. Because of the unreliability of the network, it is theoretically impossible for 100% of messages to be delivered successfully. In view of this situation, the general solution is message retransmission.

Of course, the retransmission mechanism is not unlimited retransmission, and specific retransmission strategies can be made according to the business. For example, the maximum number of retransmissions can be set to 10, and the interval between retransmissions can be increased in turn. Although this scheme is simple, the side effect is the problem of repeated delivery of messages.

Why idempotent consumption is needed

Idempotent is a mathematical conceptual theory, which means that the same operation is performed many times and once, and the final result is the same.

Take a business inappropriate but accurate chestnut: your girlfriend cheats once and cheats many times, and for you, the result is the same: you are green. So the result of cheating once and cheating many times is the same for you.

For MQ, to say the least, even if there is no problem of repeated delivery of messages in MQ, in consumer-side business, for those businesses that are sensitive to message consumption, we should also consider the idempotent consumption of messages when designing the program architecture, such as the business scenario in which users buy goods and give away red packets or credits. This scenario is very sensitive to the repeated consumption of messages, if the program is not handled properly. In the case of repeatedly sending red packets to users, it is estimated that the programmer will have to carry the pot to sacrifice to heaven again.

Idempotency is actually very easy to do.

The idempotent design of any business scenario interface should find out the data identity generated by idempotence.

From the point of view of the whole flow process of the message, the repeatability of MQ messages can be solved in two directions:

Avoid delivering repetitive messages when messages are generated, that is, message producers ensure the uniqueness of messages.

MQ itself provides filtering function for duplicate messages.

Avoid repeated consumption when messages are consumed

Image

In the first half of the process before the message is consumed, the producer can use the unique message id and ACK mechanisms to ensure that the message is delivered repeatedly, but this will greatly reduce the performance of the producer business. In general, producers need to send MQ messages asynchronously. If you need to check whether the message has been sent when sending it, this is certainly not a good design. And the inspection effect of you doing this is only to hit a very small part of the data, the loss outweighs the gain, so few people in the producer take the initiative to do the repeated delivery check of the message.

As for the interior of MQ, some MQ do provide idempotent storage designs, such as Kafka introducing Producer ID (that is, PID) and Sequence Number.

PID . Each new Producer is assigned a unique PID at initialization time, and this PID is invisible to the user.

Sequence Numbler . (for each PID, each of the data sent by the Producer corresponds to a monotonously increasing Sequence Number starting at 0.

The Broker side stores the seqnumber in the cache, and for each message received, if its sequence number is greater than 1 in the Broker cache, it will be accepted, otherwise it will be discarded. In this way, the message can be repeatedly submitted. However, a single Producer can only be guaranteed for the same Exactly Once semantics. There is no guarantee that the same Producer and a topic have different partion idempotents.

However, these are not the key points we are going to talk about today. In the actual business, the idempotent consumption of messages is also more likely to be done on the consumer side, solving the problem completely at the end of the message, whether in system design or in scalability.

As mentioned just now, since a message wants to achieve idempotent consumption, it must provide an identity for judging repetition, which can be a custom message ID or a primary key in a similar data table in which several fields in the message are combined. At present, the mainstream practice is to generate the message id according to the business characteristics, such as adding a message id to the user who gives away points for placing an order. A unique message id can be generated based on the number of userid_orderId_ credits.

With a unique message id, consumers can store the consumed message id locally to filter duplicate messages. Of course, if the amount of data is large, the historical data from a long time ago can be deleted or transferred to other backup tables. After all, the same message cannot be delivered again after a long time. The following is an example of a local message table:

Field description MsgId message idCreateTime creation time. Other useful business fields

When consuming a new message, execute the following sql statement similar to the following to get the result of whether the message has been consumed to determine whether the current message needs to be consumed repeatedly.

Select count (0) from table where MsgId=' message id'

Of course, there will be problems here. If there is only one consumer spending, there will be no problem. If multiple consumers are spending in parallel, you will need locks to ensure the ordering of the same data when judging duplicate messages. At this time, you may need distributed locks.

A solemn reminder

In addition to generating message id, there are many articles on the Internet that point out that version numbers can be used to solve idempotent problems. How many people have personally practiced this scheme? Today, we will use the case of adding points to users to solve this solution:

You need to add a version number (Version) field to the user's score table

The producer of the message adds the version number field to the message delivery

Consumers execute a specific sql of sql based on the version number of the message, similar to:

Update user set amount=amount+10, version=version+1 where userid=100 and version=1

For the repeated delivery of the same message, this can indeed achieve idempotent consumption, after all, the program uses the database locking mechanism to ensure consistency. So what's the problem?

The problem with the version number of the message

All distributed systems face the same problem, that is, the problem of data consistency, and the consumption scenario of MQ is no exception. Take the above user plus points as an example, because the producer of the message needs to query the current version number when delivering the message, similar to the following sql

Select version from table where userid=100

When the version number information is queried, the version number is delivered to MQ as part of the message body, so what happens in the case of concurrency? Suppose the current version number is 1:

Thread A queries version number 1, and then delivers a message with version number 1 and message id x. At the same time, thread B also queries the current user version with a value of 1, and then delivers a message with message id Y. At this time, whether the consumer consumes message X or message Y first, the version number of the database will increase, which leads to the consumption failure of another message due to the discrepancy of the version number.

At this point, the study on "how to use the version number to ensure the idempotency of MQ consumption messages" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.