Shulou(Shulou.com)06/03 Report--

This article mainly introduces "how RocketMQ transaction messages ensure data consistency". In daily operations, I believe many people have doubts about how RocketMQ transaction messages ensure data consistency. Xiaobian consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts of "RocketMQ transaction messages how to ensure data consistency". Next, please follow the editor to study!

Preface

In several big companies, almost every round of interviewers ("correct, almost every round of interviewers") asked the same question: is your system distributed?

A: yes.

Interviewer: so how does your distributed system solve the problem of distributed transactions? That is, how to ensure the consistency of data.

Answer: our system uses RocketMQ transaction messages to ensure the final consistency of data.

Interviewer: how can you tell me how it ensures the ultimate consistency of the data?

Answer: the answer is divided into two parts. The first part answers the implementation process of the transaction message, and the second part explains why it can ensure the ultimate consistency of the data.

The implementation process of transaction message

Transaction message

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

First, service A sends a semi-transactional message (also known as half message) to MQ. Why send a half message first? This is to ensure that the communication between Service An and MQ is normal. If there is no normal communication, Service A can directly return an exception without having to deal with the later logic.

If the half message is sent successfully, MQ receives the half message and returns a success response to service A.

After receiving the success response returned by MQ, Service A starts processing the local business logic and commits the local transaction.

If the local transaction of service An is successfully committed, a commit will be sent to MQ, indicating that the half message will be committed, and MQ will perform step 5; if the local transaction of service A fails, the local transaction will be rolled back directly and a rollback will be sent to MQ, indicating that the previous half message will be rolled back. After MQ receives the rollback message, the half message will be deleted.

If commit, the half message is written to disk.

If MQ does not receive commit or rollback messages for a long time, for example, Service An is down while processing local business, or the sent commit or rollback is lost because of weak network environment. Then MQ will try to invoke an interface provided by Service An after a certain period of time to determine the status of the half message. Therefore, the business logic of the interface provided by Service An is to judge whether the business corresponding to the previous half message has been executed successfully by the status of the corresponding data in the database. If MQ learns from this interface that the half message was executed successfully, MQ persists the half message to the local disk, and if it knows that the execution is not successful, it deletes the half message.

Service B consumes the corresponding message from the MQ.

Service B handles the local business logic and then commits the local transaction.

How to ensure the final consistency of data

The implementation process is over, maybe you have all kinds of doubts now?

Q: what is the half message?

A: it is the same as a normal message we normally send, which is stored in MQ. The only difference is that half is not immediately consumed by consumers in MQ unless the half message is commit. (as for why non-commit half messages cannot be read by consumers, this is because within MQ, transaction messages are placed in an internal queue before commit, and only commit can really put messages in topic queues that consumers can read.)

Q: why send half messages first?

A: as explained earlier, the main purpose is to ensure normal communication between Service An and MQ. If neither of them can communicate normally, then play with a hammer and return the exception directly.

Q: if MQ receives a half message, but when returning a success response, service A does not receive a success response due to network reasons, what is the phenomenon at this time?

A: when service A sends a half message, it waits for MQ to return a success response to it. If it does not receive it, service A will end directly with an exception and no further logic will be executed. Service A will not submit a commit message to MQ,MQ without receiving the commit message for a long time, so it will actively call back an interface of Service A. after querying local data through the interface, Service A will find that the business corresponding to this message has not been executed normally, then tell MQ that the half message cannot be commit. After rollback,MQ needs to know, the half message will be deleted.

Q: what if service A local transaction execution fails?

A: after service A fails to execute a local transaction, roll back its own local transaction before sending a rollback operation to MQ.

Q: what should I do with the commit or rollback message sent to MQ after the local transaction of Service A was successfully committed or failed, because the network problem was lost?

A: like the previous question, MQ has not received commit or rollback messages of half messages for a long time. MQ will actively call back the interface of Service A to determine what to do with this half message.

Q: what I said above is all about the implementation process of transaction messages. What does this have to do with how transaction messages ensure the final consistency of data?

A: it does. First, service An executes a local transaction and commits and sends messages to MQ, which are two write operations, and then through RocketMQ's transaction messages, we ensure that either both writes succeed or fail. Then let other systems, such as Service B, consume messages in MQ, and then execute their own local transactions, so that in the end, are the data states of Service An and Service B consistent? This is what ultimate consistency means.

If the data status of service An and service B is required, the two are consistent between service An and the client, which is strong consistency, and RocketMQ cannot guarantee strong consistency.

At present, "reliable information to ensure the ultimate consistency of data" is a scheme adopted by many large factories, basically through MQ and compensation mechanism to ensure data consistency. (the so-called reliable information, that is, the message is not lost, how to ensure that the MQ message is not lost, the next article will be written, which is also a common question in the interview.)

Q: service B local transaction commit failed, what should I do?

A: if service B's local transaction commit fails, you can retry several times until it succeeds. If the submission fails after many retries, for example, if the DB corresponding to service B goes down, as long as service B does not submit the offset of this message to MQ. If the offset is not committed, MQ will continue to push the message to service B after a certain period of time, and service B can continue to execute the local transaction and commit until it is successful. In this way, the final consistency of service An and service B data is still guaranteed.

Code implementation

Transaction messages that use RokcetMQ involve two main parts:

How to send semi-transactional messages can be achieved through the "TransactionMQProducer" class.

TransactionMQProducer transactionMQProducer = new TransactionMQProducer ("producerGroup"); TransactionSendResult result = transactionMQProducer.sendMessageInTransaction (msg, null); / / determine whether the half message was sent successfully by result if (result.getSendStatus () = = SendStatus.SEND_OK) {/ / success} else {/ / failure}

We mentioned earlier that Service A needs to provide an interface for MQ to call back Service A. in fact, this interface is a listener: "TransactionListener" method. This is an interface that provides two methods.

Public interface TransactionListener {/ / when the half message is sent successfully, we implement our own business logic here, and then commit or rollback gives MQ LocalTransactionState executeLocalTransaction (final Message msg, final Object arg) / / this method is the method for MQ callback. MQ judges the status of the half message by calling back this method / / you can see that the parameter of this method is MessageExt, that is, the content of the half message. According to MessageExt, we can completely judge whether the previous business has been successfully processed LocalTransactionState checkLocalTransaction (final MessageExt msg) in Service A;}

In actual use, we need to implement this interface, for example:

Public class MyTransactionListener implements TransactionListener {@ Override public LocalTransactionState executeLocalTransaction (Message msg, Object arg) {try {/ / processing business logic / /.... / / Business logic processing succeeded, commit return LocalTransactionState.COMMIT_MESSAGE;} catch (Exception e) {} / / Business processing failed, rollback return LocalTransactionState.ROLLBACK_MESSAGE;} @ Override public LocalTransactionState checkLocalTransaction (MessageExt msg) {return null;}}

In addition, when creating the producer, specify the listeners that we implement

TransactionMQProducer transactionMQProducer = new TransactionMQProducer ("producerGroup"); transactionMQProducer.setTransactionListener (new MyTransactionListener ()); at this point, the study of "how RocketMQ transaction messages ensure data consistency" is over, hoping to solve everyone's doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.