Shulou(Shulou.com)06/01 Report--

This article mainly introduces the example analysis of flexible transaction consistency in apache rocketmq, which has a certain reference value. Interested friends can refer to it. I hope you can learn a lot after reading this article.

First, best effort notification

TCC segmented submission is suitable for business scenarios with high consistency and real-time requirements in distributed architecture, and there are also low real-time services in actual business, such as common SMS notification, client message, operation system update and other services. In order to reduce the complexity and pressure of the core process, best effort notification can be adopted to achieve flexible transaction management.

For example, in common third-party payment services, message notifications are generated after the completion of local business and payment end business processing. The basic process is as follows:

After the preprocessing of local business is completed

Request third party payment service

The payment operation successfully sent a message to the account.

Payment service callback local business

Local service generation system notification message

There are some basic features in the message scenario of the above process. After the core business processing is completed, a message notification is sent to allow failure, and message loss is allowed within a specified period of time or after a specified number of retries, that is, the unreliability of the message.

In the actual payment system, when the daily reconciliation is started, the flow of the day will be checked. If it is found that there is an unfinished process of the payment flow, there will be a state to make up for it, and subsequent processing can be continued. This method is very commonly used in the reconciliation.

Second, reliable information

Distributed transaction is based on the implementation scheme of the final consistency of reliable messages. Since it is a reliable message, it requires that MQ must support transaction management, so as to ensure the consistency before and after the business.

1. RocketMQ transaction message

RocketMQ began to support distributed transaction messages in version 4.3. it uses the idea of 2PC to implement commit transaction messages, and adds a compensation logic to handle two-phase timeout or failure messages, as shown in the following figure:

The figure above illustrates the general scheme of the transaction message, which is divided into two processes: the sending and submission of the normal transaction message and the compensation process of the transaction message.

1.1 send and submit

(1) send messages (half messages, that is, send but not be consumed)

(2) the server response message is written into the result

(3) execute the local transaction according to the sending result. If the write fails, the half message is not visible to the business and the local logic is not executed.

(4) execute Commit or Rollback according to the status of the local transaction (the Commit operation generates the message index, and the message is visible to the consumer)

1.1 compensation process

(1) initiate a "backcheck" from the server for transaction messages without Commit/Rollback (messages of pending status)

(2) Producer receives the retrace message and checks the status of the local transaction corresponding to the retrace message.

(3) re-Commit or Rollback according to the local transaction status

The compensation phase is used to solve the case of timeout or failure of message Commit or Rollback.

1.3 Design principles

In the main flow of RocketMQ transaction messages, how the first phase of the message is not visible to the user. Among them, the most important feature of transaction messages compared with ordinary messages is that the messages sent in one phase are invisible to users. So how do you write a message but not be visible to the user? The practice of RocketMQ transaction messages is that if the message is a half message, the topic of the original message and the message consumption queue will be backed up, and then the subject will be changed to RMQ_SYS_TRANS_HALF_TOPIC. Because the consumer group does not subscribe to this topic, the consumer cannot consume messages of half type. Then RocketMQ will start a scheduled task to pull messages from Topic for RMQ_SYS_TRANS_HALF_TOPIC for consumption. According to the producer group, a service provider will send a request to check the transaction status, and decide whether to commit or roll back the message according to the transaction status.

2. Final consistency

Based on the above characteristics of the reliability of RocketMQ transaction messages, the final consistency of transactions under a certain type of business can be achieved. Message delivery consistency means that the business action that generates the message is consistent with the message delivery, that is, if the business operation is successful, the asynchronous message generated by the business operation must be sent, otherwise the business failure will be rolled back and the message will be discarded.

The process is basically as follows:

Send a half transaction message, which cannot be consumed

Local business code logic processing completed

Send a confirmation message indicating that the message can be consumed

If the message producer is abnormal, cancel the overall action

This process is mainly aimed at the message producer, and in the actual development, the consumer of the message is also very difficult to deal with. To ensure the final consistency, there must be a problem: the consumer is abnormal and the message is constantly retried. There may be some business processing success and some business processing failure. At this time, it is necessary to solve the idempotent problem of the service interface.

3. Brief introduction of idempotent interface 1 and idempotent

The characteristic of an idempotent operation in programming is that the effect of arbitrary execution is the same as that of one execution. That is, requesting a resource once and multiple times will have the same effect.

In complex asynchronous processes, pay special attention to the problem of failed retry. Usually, in the payment process, every time the interface is requested, each step of the data update operation will be preceded by the process of status query to determine whether the next data update should be performed.

2. Idempotent interface

In the system service interface request, any explicit interface response, such as failure or success, makes the business process easy to handle, but for example, in the payment scenario, if the request times out, how to judge the result status of the service: client request timeout, local service timeout, request payment timeout, payment callback timeout, client response timeout, etc., or based on the continuous retry mechanism of MQ, under some business abnormal conditions. If no success is returned, the message will always retry.

This requires the design of procedural state management, especially under the message retry mechanism, heavy transaction control is rarely used for retried business interfaces. Some services are executed and only one state needs to be judged. The next time the message is retried, skip it. You only need to compensate the unprocessed business. Under the retry mechanism, the message will always retry until some of the businesses have been executed successfully. Until it's all finished.

Thank you for reading this article carefully. I hope the article "sample Analysis of flexible transaction consistency in apache rocketmq" shared by the editor will be helpful to you. At the same time, I also hope you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.