Shulou(Shulou.com)06/02 Report--

Say in front.

Apache RocketMQ-4.3.0 officially Release the features of transaction messages, following this recent hot spot. In the first article, let's talk about a long-standing problem in software engineering-distributed transactions (Distributed Transaction).

This technology has also been widely implemented, utilized and optimized in various big companies such as Ali and Tencent. However, because there are difficulties in theory, distributed transactions have become a technical barrier for large companies to small ones. I believe that the students who read this article must have heard a lot of terms about distributed transactions, comparing two-phase commit, TCC, final consistency, etc., so there are not many popular concepts here.

Distributed transaction based on RocketMQ

Let's get to the point and use RocketMQ to design our own distributed transaction components.

Give an example of a virtual scene that leads to problems.

Users transfer 100 yuan from Agricultural Bank to China Merchants Bank. The systems of Agricultural Bank and China Merchants Bank are respectively deployed in their own computer room. The systems communicate with each other through messages to prevent excessive coupling.

The whole model can be improperly described as: after the Agricultural Bank deducted 100 yuan, sent a "deducted" message to China Merchants Bank. China Merchants Bank received the message that the deduction was successful, and then added 100 yuan to the account of China Merchants Bank.

The problem is, on the side of Agricultural Bank of China, plan 1. Deduct 100 yuan before you send the message, plan 2. Send the message first and then deduct 100 yuan.

Sort out the scenario where the whole transaction is inconsistent:

Option 1

The Agricultural Bank was successful after deducting 100, but the message failed, and China Merchants Bank did not add 100.

Option 2

The message was sent successfully, but the Agricultural Bank failed to deduct 100 yuan. China Merchants Bank received the message by 100.

Students should have found the problem, deduction and sending messages these two things, there is no way to change the order to achieve "simultaneous success" or "simultaneous failure". If the former succeeds and the latter fails, it will lead to inconsistency.

RocketMQ, hereinafter referred to as RMQ, introduces a new message type in order to implement transaction messages: TransactionMsg

A complete transaction message is divided into two parts:

HalfMsg (Prepare) + Commit/RollbackMsg

After Producer sends HalfMsg, because HalfMsg is not a complete transaction message, Consumer cannot consume the message immediately. Producer can Commit or Rollback the HalfMsg to terminate the transaction (EndTransacaction). Only when Commit has HalfMsg can Consumer consume this message. RMQ periodically asks Producer if it is possible to end their lifecycles by Commit or Rollback HalfMsg that are not terminated due to errors in order to reach a final agreement on the transaction.

Still in the transfer scenario, we use RMQ transaction messages to optimize the process:

Agricultural Bank of China sends HalfMsg synchronously to RMQ, with the message that Agricultural Bank is about to deduct RMB100.

After the HalfMsg of Agricultural Bank of China is successfully sent, it executes the local transaction of the database and deducts 100 yuan from its own system.

Agricultural Bank of China to check the execution of local affairs

Local transaction returned successfully, Agricultural Bank of China submitted (Commit) HalfMsg to RMQ

China Merchants Bank system subscribed to RMQ and successfully received the information that Agricultural Bank had deducted 100 yuan.

China Merchants Bank system performs local affairs, adding 100 yuan to China Merchants Bank system

Cdn.xitu.io/2018/9/26/166159d6de987c5e?w=1080&h=834&f=jpeg&s=49401 ">

Figure 1:RMQ transaction message principle

Similarly, let's analyze whether there will be inconsistencies in the process one by one:

HalfMsg sent by Agricultural Bank of China is sent synchronously (Sync). If HalfMsg is not sent successfully, local transactions will not be executed at all.

The HalfMsg was sent successfully, but the Agricultural Bank deducted money * * the local transaction failed. If the local transaction is not successful, immediately send Rollback to roll back HalfMsg**. Just pretend nothing happened before.

The local transaction of Agricultural Bank of China was successful, but Commit failed, but because HalfMsg is already in RMQ, RMQ can let Agricultural Bank of China re-check whether the local transaction is successful and re-Commit through the timing program. It is the same reason that Rollback failed.

After China Merchants Bank consumes the message, the local transaction fails, but the message received by China Merchants Bank is persisted in MQ and can even be persisted in China Merchants Bank database. Transaction retry can be carried out.

The case just discussed is very idealized, the whole distributed transaction only involves the change of amount, but in the real online system, the local transaction as the sender of the message may be very complex and may involve dozens of different tables, so RMQ uses timers to Check HalfMsg, can it check whether the data of each table involved in the transaction has been submitted successfully? Obviously this kind of solution is very business intrusive, and it is very difficult to be componentized. Therefore, it is necessary to design a Transaction table in the local transaction to bind the business table and the Transaction in the same local transaction. If the Agricultural Bank's deduction local transaction is successful, the status of the TransactionId should have been recorded in Transaction as "completed". When you finally need to check, you only need to check whether the status of the corresponding TransactionId is "completed", regardless of the specific business data.

Let's talk about one small detail.

Careful students may find that the discussion of No.3 is actually a bit lax. When RMQ calls Commit or Rollback, it uses the way of Oneway. If you are familiar with the RMQ source code, you know that this kind of network call only sends Request one-way and will not get Response. There is a great improvement in message sending performance, but Producer will not know if it really fails, and the transaction can only be terminated by checking the HalfMsg regularly.

Public void endTransactionOneway (final String addr, final EndTransactionRequestHeader requestHeader, final String remark, final long timeoutMillis) throws RemotingException, MQBrokerException, InterruptedException {RemotingCommand request = RemotingCommand.createRequestCommand (RequestCode.END_TRANSACTION, requestHeader); request.setRemark (remark); / / use Oneway to send this.remotingClient.invokeOneway (addr, request, timeoutMillis) of type end transaction;} distributed transactions separated from RocketMQ

Not all MQ can support transactional messages. How to use general MQ to build distributed transactional components or even abstract into a transactional SOA service?

In fact, if we take a closer look at the transaction message of RMQ, we can break it into two parts:

Transaction Manager + messages

The so-called transaction manager is the management of transaction preparation (Prepare), commit (Commit) and rollback (Rollback). In addition, it also includes a timing checker for preparatory transactions.

Message, which refers to the general synchronous message, can clearly get the sending result after being sent, which can be used to decouple the transaction system from the business system. Almost all distributed MQ supports this kind of message.

Let's design our own DistributedTransaction SOA, hereinafter referred to as DT-SOA.

Figure 2: distributed transaction service

The process remains the same, but distributed transactions are no longer strongly dependent on RMQ, but are replaced by generic MQ:

When system A sends a transaction, first call the Prepare method of DT-SOA to start the transaction. Since it is a synchronous call, get the SendResult. If the transaction is sent successfully, get the ID--TID of the global distributed transaction.

System A uses the acquired TID to execute the local transaction, and the local transaction contains the Transaction status table. After success, the corresponding state of the TID is set to "completed".

System A calls DT-SOA to commit the transaction, and DT-SOA uses MQ to send synchronization messages to system B.

System B listens to the corresponding Topic and executes the corresponding local transaction after receiving the message.

For more wonderful articles, please follow my official Wechat account: Eric's technology.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.