Shulou(Shulou.com)06/01 Report--

Not long ago, I transferred 10,000 yuan from Alipay to Yu'ebao, which is a common thing in daily life, but as an occupational disease of Internet R & D personnel, I thought about what to do if the system failed after deducting 10,000 from Alipay. At this time, the Yu'e Bao account did not increase by 10,000, and the data will be inconsistent.

Similar shadows can be found in all types of systems in the above scenarios. For example, in e-commerce systems, when a user places an order, in addition to inserting a record in the order table, the number of goods in the corresponding commodity table must be reduced by 1. How to guarantee?! In the search advertising system, when users click on an advertisement, in addition to adding a record in the click event table, they also have to go to the merchant account table to find the merchant and deduct the advertising fee, how to guarantee it?! Wait, wait, wait. I believe everyone will encounter a similar situation more or more.

In essence, the problem can be abstracted as: when the data of one table is updated, how to ensure that the data of another table must be updated successfully.

1 Local transaction

Or take Alipay transfer Yu'e Bao as an example, suppose there are

Alipay account form: a (id,userId,amount)

Yu'e Bao account form: B (id,userId,amount)

User's userId=1

The transfer of 10,000 yuan from Alipay to Yu'ebao is divided into two steps:

1) pay treasure watch to deduct 10, 000: update A set amount=amount-10000 where userId=1

2) increase the balance by 10, 000: update B set amount=amount+10000 where userId=1

How to ensure the balance of Alipay Yu'e Bao?

Some people say that this is very simple and can be solved by transaction. download

one

two

three

four

five

Begin transaction

UpdateAset amount=amount-10000where userId=1

UpdateBset amount=amount+10000where userId=1

Endtransaction

Commit

Quite right, if you use spring, you can do the above transaction function with a single comment.

Java

one

two

three

four

five

@ Transactional (rollbackFor=Exception.class)

Publicvoidupdate () {

UpdateATable (); / / Update Table A

UpdateBTable (); / / Update Table B

}

If the system scale is small and the data tables are on a database instance, the above local transaction mode can run well, but if the system scale is large, for example, Alipay account table and Yu'ebao account table are obviously not on the same database instance, they are often distributed on different physical nodes, and the local transaction has lost the opportunity to display its ability.

Now that the local transaction fails, the distributed transaction naturally takes the stage.

2 distributed transaction-two-phase commit protocol download

The two-phase commit protocol (Two-phase Commit,2PC) is often used to implement distributed transactions. It is generally divided into two roles: coordinator C and several transaction executors Si, where the transaction executor is the specific database, and the coordinator can be on the same machine with the transaction executor.

1) our application (client) initiates an initial request to TC

2) TC first writes the message to the local log, and then initiates the message to all Si. Take Alipay transfer to Yu'e Bao as an example, the prepare message from TC to An is to inform Alipay database of the corresponding account deduction of 10,000, and the prepare message from TC to B is to inform Yu'ebao database that the corresponding account is increased by one week. Why do you need to write the local log before performing the task, mainly for recovery after the failure? the local log acts as a credential in real life. If there is no local log (voucher), it is easy to die without proof.

3) after receiving the message, Si executes a specific native transaction, but does not commit. If it returns successfully, it will not return successfully. By the same token, the message to be returned should be written in the log as a credential before returning.

4) TC collects messages returned by all executors, and if all actuators return yes, then send commit messages to all actuators, and the executors perform local transaction commit operations after receiving commit; if any actuator returns no, then send abort messages to all actuators, and the executors perform transactional abort operations after receiving abort messages. download

Note: TC or Si write messages sent or received to the log first, mainly for recovery after failure. If a Si recovers from a failure, check the local log first, if it has been received, submit it, and if it is rolled back. If so, ask TC again to determine the next step. If there is nothing, it is likely that the Si crashed in the phase, so it needs to be rolled back.

However, students who have used the above two-phase submission will find that the performance is too poor to be suitable for high concurrency systems at all. Why?

1) two-phase commit involves network communication between multiple nodes, and the communication time is too long!

2) the transaction time is relatively longer, and the locked resource time is also longer, resulting in a lot of resource waiting time!

It is precisely because of the serious performance problems of distributed transactions that most of the highly concurrent services are avoided and often solve the problem of data consistency in other ways.

3 use message queuing to avoid distributed transaction downloads

If we take a closer look at life, many scenes of life have given us hints.

For example, after the famous Yaoji fried liver in Beijing ordered and paid for the fried liver, they will not directly give you the fried liver you ordered, but give you a small ticket, and then let you take the small ticket to the shipping area to wait in line to pick it up. Why do they separate the act of paying and picking up goods? There are many reasons, one of which is to enhance their reception capacity (higher concurrency).

Back to our question, as long as this small ticket is there, you can finally get fried liver. The same is true of the transfer service. When 10,000 is deducted from the Alipay account, we only need to generate a voucher (message) that says "increase the Yu'e Bao account by 10,000". As long as the voucher (message) can be reliably preserved, we can finally increase the Yu'e Bao account by 10,000 with this voucher (message), that is, we can rely on this voucher (message) to complete the final consistency.

3.1 how to reliably save credentials (messages)

There are two ways: download

3.1.1 the way services are coupled with messages

While completing the deduction, Alipay records the message data, which is stored in the same database instance as the business data (the message record table is named message).

one

two

three

four

five

Begin transaction

UpdateAset amount=amount-10000where userId=1

Insert into message (userId,amount,status) values (1m 10000)

Endtransaction

Commit

The above affairs can guarantee that as long as the money is deducted from the Alipay account, the news will be preserved.

When the above transaction is successfully submitted, we notify Yu'e Bao through the real-time message service. After Yu'e Bao is processed successfully, a successful reply message is sent, and Alipay deletes the message data after receiving the reply.

3.1.2 download by decoupling business and message

The above way of saving messages makes the message data and business data tightly coupled, which is not elegant enough from the perspective of architecture, and is easy to cause other problems. To decouple, you can use the following ways.

1) before the debit transaction is submitted, Alipay requests to send a message to the real-time messaging service. The real-time messaging service only records the message data, but does not actually send it. Only after the message is successfully sent will the transaction be submitted.

2) when the Alipay deduction transaction is successfully submitted, confirm and send it to the real-time messaging service. The real-time message service actually sends the message only after the confirmation instruction is received.

3) when the Alipay debit transaction fails to be rolled back, it will be canceled to the real-time messaging service. After receiving the cancellation instruction, the message will not be sent

4) for those unconfirmed messages or canceled messages, it is necessary to have a message status confirmation system to regularly go to Alipay to query the status of the message and update it. Why this step is needed, for example: suppose that after the Alipay deduction transaction in step 2 is successfully submitted, the system hangs, and the message status is not updated to "confirm sending", resulting in the message cannot be sent.

Advantages: independent storage of message data, reducing the coupling between business system and message system

Disadvantages: two requests are required for one message to be sent; the business processing service needs to implement the message status check interface.

3.2 how to solve the problem of repeated delivery of messages

Another serious problem is the repeated delivery of messages. Take our Alipay transfer to Yu'e Bao as an example, if the same message is repeated twice, then our Yu'e Bao account will increase by 20,000 instead of 10,000.

Why is the same message repeated? For example, after Yu'e Bao processes the message msg, it sends the message successfully to Alipay. Normally, Alipay should delete the message msg, but if Alipay tragically dies at this time, as soon as it sees that the message msg is still there after restart, it will continue to send message msg.

The solution is very simple. Add a message application status table (message_apply) to Yu'e Bao. In popular terms, it is a ledger to record the consumption of messages. Each time a message is sent, before the actual execution, go to the message application status table to query it. If the description is a duplicate message, discard it. If it is not found, it will be executed, and insert it into the message application status table (the same transaction).

one

two

three

four

five

six

seven

eight

Foreachmsg inqueue

Begin transaction

Select count (*) ascnt from message_apply where msg_id=msg.msg_id

Ifcnt==0then

UpdateBset amount=amount+10000where userId=1

Insert into message_apply (msg_id) values (msg.msg_id)

Endtransaction

Commit

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.