Shulou(Shulou.com)06/01 Report--

This article will explain in detail the solution to the consistency of writing database and sending mq messages at the same time. The content of the article is of high quality. Therefore, Xiaobian shares it with you as a reference. I hope you have a certain understanding of relevant knowledge after reading this article.

I. Introduction

With the growth of the company's business, it is difficult for the single application architecture to meet the rapid iteration and performance requirements of the business, and the service transformation will be carried out to split the original huge single application into different services according to the business and other elements. Then before the service transformation, the first thing to face is the technology selection of the service infrastructure, the most important of which is the communication middleware between the services. Communication between services can be divided into synchronous mode and asynchronous mode. Synchronous mode is represented by RPC, asynchronous mode will generally choose mq.

II. Practical significance of the problem

If we want to use message queuing in a service split, what problems do we need to solve? First of all, Qunar.com provides online booking service for tourism products, so it involves e-commerce transactions, in which we believe that data consistency is a very critical element. Then our MQ must provide consistency guarantees.

MQ provides consistency assurance in two ways. When sending messages, how can we ensure that the business operation and message sending are consistent, that is, the business operation success message is not sent or the message is sent but the business is not successful? For example, if the payment service uses a message to notify the ticketing service, then payment cannot be successful, but the message is not sent, which will cause user complaints; but payment cannot be unsuccessful, but the message is finally issued, which will cause losses to the company. To sum up, messaging and business needs to have transaction guarantees. At the other end of the consistency spectrum are consumers, such as temporary consumer errors or network failures, and how do we ensure that messages are eventually processed? Then we achieve final consistency by consuming ACKs and retries.

III. Using database transactions to solve consistency problems

When you think of consistency, you think of transactions, and when you think of transactions, you think of relational databases, so can we use the tried and tested transactions in relational DB to achieve this consistency? Let's take MySQL as an example. For db in the same instance of MySQL, if they share the same Connection, they can be in the same transaction. Take the following figure as an example, we have a MySQL instance listening on port 3306, and then there are two DBs A and B on the instance, so the following pseudocode can run in the same transaction.

With this layer of assurance, we can transparently implement business operations and message sending in the same transaction. First, we initialize a message db in all MySQL instances of the company, which can be put into the automation process (it is said that it is completed by the operation and maintenance team in Qunar), transparent to the application. Then we just need to put messaging and business operations into the same DB transaction.

Let's look at an actual scenario. In the payment scenario, after the payment is successful, we need to insert a payment stream and send a payment completion message to notify other systems. Then inserting payment flow and sending message need to be consistent, any step that is not successful will eventually lead to problems. Then there is the following code

The above code can be interpreted with the following pseudocode

In fact, when producer.sendMessage is executed, the message is not sent out through the network, but only a record is inserted into the message database on the same instance of the service DB, and then the callback of the transaction is registered. After the transaction is actually submitted, the message is sent out from the network. At this time, if the message is successfully sent to the server, it will be deleted immediately. If the message fails to be sent, the message stays in the message store. At this time, we have a compensation task to pull the message from the message store and resend it until it is successfully sent. The whole process is shown below

1, begin tx Start local transaction

2, do work to perform business operations

3. insert message Insert a message into the same instance message library

4. end tx transaction submission

5, send message The network sends a message to the server

6, response server response message

Delete message If the server replies successfully, delete the message.

8, scan messages compensation task scan no message sent

9, send message compensation task compensation message

10. delete messages Compensation task Delete compensation success messages

About writing database and sending mq message transaction consistency solution to share here, I hope the above content can be of some help to everyone, you can learn more knowledge. If you think the article is good, you can share it so that more people can see it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.