Shulou(Shulou.com)06/02 Report--

Or you may tell me that there is no transaction when you have already deducted the money from your online shopping. This series of situations are caused by the absence of transactions. This shows some of the importance of affairs in life.

When there is a business, you go to the store to buy things, that is to pay money and deliver goods on the other. With business, you go online shopping, deduction will generate order transactions.

Specific definition of transaction

Transactions provide a mechanism to integrate all operations involved in an activity into an indivisible execution unit, and all operations that make up a transaction can be committed only if all operations can be executed normally, as long as any of them fails, will cause the entire transaction to be rolled back.

Simply put, transactions provide a "do nothing or All or Nothing" mechanism.

Database local transaction

ACID

When it comes to database transactions, we have to say that there are four major features of database transactions: ACID:

A: Atomicity, all operations in a transaction (transaction) are either completed or not completed, and do not end at some point in the middle.

An error occurs during the execution of a transaction and is Rollback back to its state before the transaction starts, as if the transaction had never been executed.

Just like when you buy something, you either pay the money and receive the goods together, or if you can't deliver the goods, you will refund the money.

C: Consistency, the consistency of a transaction means that the database must be in a consistent state before and after a transaction is executed.

If the transaction completes successfully, all changes in the system will be applied correctly and the system will be in a valid state.

If an error occurs in the transaction, all changes in the system are automatically rolled back and the system returns to its original state.

I: Isolation, which means that in a concurrent environment, when different transactions manipulate the same data at the same time, each transaction has its own complete data space.

Changes made by concurrent transactions must be isolated from changes made by any other concurrent transactions. When a transaction views data updates, the state of the data is either before another transaction modifies it, or after another transaction modifies it, and the transaction does not view the data in the intermediate state.

For example, the fact that you buy something does not affect other people.

D: Durability, which means that as long as the transaction ends successfully, its updates to the database must be saved permanently.

Even if a system crash occurs, the database can be restored to the state it was when the transaction ended successfully after restarting the database system.

For example, when you buy something, you need to record it in the ledger, even if the boss forgets it.

Principle of InnoDB implementation

InnoDB is a storage engine of MySQL, and most people are familiar with MySQL. Here is a brief introduction to some of the basic principles of database transaction implementation.

In a local transaction, services and resources can be considered as one under the transaction package, as shown in the following figure:

Our local transactions are managed by the resource manager:

The ACID of the transaction is guaranteed by InnoDB logs and locks. Transaction isolation is achieved through database locking mechanism, persistence is achieved through Redo Log (redo log), atomicity and consistency are achieved through Undo Log.

The principle of Undo Log is simple: in order to satisfy the atomicity of the transaction, before manipulating any data, first backup the data to a place (the place where the backup of the data is called Undo Log). Then modify the data.

If an error occurs or the user executes the Rollback statement, the system can use the backup in Undo Log to restore the data to the state it was before the transaction began.

In contrast to Undo Log, Redo Log records a backup of new data. Just persist the Redo Log before the transaction is committed, and there is no need to persist the data.

When the system crashes, the data is not persisted, but the Redo Log is persisted. The system can restore all data to the latest state according to the content of Redo Log. Students who are interested in the specific implementation process can search and expand on their own.

Distributed transaction

What is a distributed transaction

Distributed transaction means that the participants of the transaction, the server that supports the transaction, the resource server and the transaction manager are located on different nodes of different distributed systems.

To put it simply, a large operation consists of different small operations, which are distributed on different servers and belong to different applications. Distributed transactions need to ensure that these small operations either succeed or fail.

In essence, distributed transaction is to ensure the data consistency of different databases.

The causes of distributed transactions

Judging from the above local affairs, we can be divided into two parts:

Service generates multiple nodes

Resource generates multiple nodes

Service multiple nodes

With the rapid development of the Internet, service architecture models such as micro-services and SOA are being used on a large scale.

To take a simple example, within a company, the user's assets may be divided into many parts, such as balances, points, coupons, and so on.

Within the company, it is possible that the points function is maintained by a micro-service team, while coupons are maintained by another team.

In this way, there is no guarantee that the coupon will be deducted successfully after the deduction.

Resource multiple nodes

Similarly, the Internet is developing so fast that our MySQL generally has to carry out sub-databases and tables when loading tens of millions of data.

For an Alipay transfer business, if you transfer money to a friend, it is possible that your database is in Beijing and your friend's money is in Shanghai, so we still can't guarantee that they can succeed at the same time.

The foundation of distributed transactions

From the above point of view, distributed transactions came into being with the rapid development of the Internet, which is inevitable.

We have mentioned before that the four ACID features of the database can no longer meet our distributed transactions, and at this time some new bosses put forward some new theories.

CAP

CAP theorem, also known as Brewer theorem. For architects who design distributed systems (not just distributed transactions), CAP is your introductory theory.

C (consistency): for a given client, the read operation returns the latest write operation.

For data distributed on different nodes, if the data is updated in one node, then if the latest data can be read in other nodes, it is called strong consistency. If a node does not read it, it is distributed inconsistency.

A (availability): non-failed nodes return reasonable responses (not errors and timeouts) within a reasonable period of time. One of the two keys to usability is reasonable time and the other is reasonable response.

A reasonable time means that the request cannot be blocked indefinitely and should be returned at a reasonable time. A reasonable response means that the system should explicitly return the results and the results are correct, which means, for example, that 50 should be returned instead of 40.

P (partition fault tolerance): when there is a network partition, the system can continue to work. For example, there are multiple machines in the cluster, and there is a problem with the network of machines, but the cluster can still work properly.

Anyone familiar with CAP knows that the three cannot be shared. If you are interested, you can search for the proof of CAP. In a distributed system, the network is not 100% reliable, and partitioning is an inevitable phenomenon.

If we choose CA instead of P, then when partitioning occurs, in order to ensure consistency, the request must be rejected, but A does not allow it, so it is theoretically impossible for distributed systems to choose CA architecture, only CP or AP architecture.

For CP, to abandon usability and pursue consistency and partition fault tolerance, our ZooKeeper is actually the pursuit of strong consistency.

For AP, abandon consistency (consistency here is strong consistency), pursue partition fault tolerance and availability, this is the choice of many distributed system design, the following BASE is also extended according to AP.

By the way, CAP theory ignores network delay, that is, when a transaction is committed, there is no delay in copying from node A to node B, but in reality this is obviously impossible, so there is always a certain amount of time inconsistency.

At the same time, choosing two of the CAP, for example, if you choose CP, is not asking you to give up A. Because the probability of P occurrence is so small, you still need to guarantee CA most of the time.

Even if the partition appears, you have to prepare for the later A, for example, through some logging means, other machines are restored to be available.

BASE

BASE is the abbreviation of three phrases: Basically Available (basic available), Soft state (soft state), and Eventually consistent (final consistency). It is an extension of AP in CAP.

Basic availability: when a failure occurs in a distributed system, it is allowed to lose some of the available functions to ensure the availability of core functions.

Soft state: allows an intermediate state in the system that does not affect system availability, referring to inconsistencies in CAP.

Final consistency: final consistency means that after a period of time, all node data will be consistent.

BASE solves the problem that there is no network delay in the theory of CAP, and uses soft state and final consistency in BASE to ensure the consistency after delay.

BASE is the opposite of ACID, which is completely different from the strong consistency model of ACID, but achieves availability at the expense of strong consistency, and allows data to be inconsistent for a period of time, but eventually reach a consistent state.

Distributed transaction solution

With the above theoretical basis, here are several common solutions for distributed transactions.

Do you really want to distribute transactions?

Before you talk about the solution, you must first make it clear whether you really need distributed transactions.

There are two reasons for distributed transactions mentioned above, one of which is that there are too many microservices. I've seen too many teams maintain a few microservices alone, and too many teams are so overdesigned that they make everyone tired.

If there are too many micro-services, it will lead to distributed transactions. At this time, I will not suggest you to adopt any of the following solutions, but please aggregate the micro-services that require transactions into a stand-alone service and use the local transactions of the database.

Because any solution will increase the complexity of your system, the cost is too high, never introduce unnecessary cost and complexity because of the pursuit of certain designs.

If you are sure you need to introduce distributed transactions, take a look at the following common scenarios.

2PC

When it comes to 2PC, you have to talk about XA Transactions in database distributed transactions.

There are two phases in the XA protocol:

The transaction manager requires each database involved in the transaction to pre-commit (precommit) this operation and reflects whether it can be committed.

The transaction coordinator requires each database to commit or roll back the data.

Advantages:

As far as possible to ensure the strong consistency of data, low cost, in the major major databases have their own implementation, for MySQL is from 5.5 to start to support.

Disadvantages:

Single point of problem: the role of the transaction manager in the entire process is critical. If the transaction manager goes down, such as when the first phase is completed, or when the transaction manager is ready to commit in the second phase, the resource manager will block all the time, causing the database to become unusable.

Synchronous blocking: after it is ready, the resources in the resource manager remain blocked until the commit is complete and the resources are released.

Data inconsistency: although the two-phase commit protocol is designed for distributed data strong consistency, there is still the possibility of data inconsistency.

For example, in the second phase, suppose the coordinator sends out the transaction Commit notification, but because of the network problem, only some participants receive the notification and perform the Commit operation, while the rest of the participants remain blocked because they do not receive the notification, which leads to data inconsistency.

Generally speaking, the XA protocol is relatively simple and low-cost, but its single point problem and inability to support high concurrency (due to synchronization blocking) are still its biggest weaknesses.

TCC

The concept of TCC (Try-Confirm-Cancel) was first put forward by a paper called "Life beyond Distributed Transactions:an Apostate's Opinion" published by Pat Helland in 2007.

Compared with the XA described above, the TCC transaction mechanism addresses the following shortcomings:

The single point of coordinator is solved, and the business activity is initiated and completed by the main business party. The business activity manager also becomes multipoint, introducing clusters.

Synchronous blocking: the timeout is introduced and compensated after the timeout, and the entire resource is not locked, the resource is converted into business logic form, and the granularity becomes smaller.

Data consistency, with the compensation mechanism in place, is controlled by the business activity manager.

The explanation for TCC:

Try phase: try to execute, complete all business checks (consistency), reserve the necessary business resources (quasi-isolation).

Confirm phase: confirm the real execution of the business, do not make any business check, only use the business resources reserved in the Try phase, and the Confirm operation is idempotent. Idempotent design is required, and retry is required after Confirm failure.

Cancel phase: cancel the execution, release the business resources reserved in the Try phase, and the Cancel operation is idempotent. The exception handling scheme of Cancel phase is basically the same as that of Confirm phase.

Take a simple example: if you buy a bottle of water for 100yuan, the Try phase: you need to check your wallet to see if it is enough and lock it. The water is the same.

If there is a failure, Cancel (release the 100RMB and the bottle of water), and if Cancel fails, retry Cancel regardless of failure, so you need to remain idempotent.

If all are successful, Confirm, confirm that the 100RMB is deducted, and the bottle of water is sold, and if Confirm fails, no matter what fails, try again (depending on the activity log for retry).

It is suitable for TCC:

Strong isolation, strict consistency requirements of the activity business.

Business that takes a short time to execute.

Implementation reference: https://github.com/liuyangming/ByteTCC/.

Local message table

The scheme of local message table was originally put forward by eBay and the complete scheme of eBay https://queue.acm.org/detail.cfm?id=1394128.

The core of this scheme is to asynchronously execute tasks that require distributed processing through message logs. Message logs can be stored in local text, database, or message queues, and retried automatically or manually through business rules.

Manual retry is more applied to the payment scenario, through the reconciliation system to deal with post-event problems.

For local message queues, the core is to transform large transactions into small transactions. Or take the above example of using 100 yuan to buy a bottle of water.

1. When you deduct money, you need to add a new local message table to the server where you deduct the money, and you need to write your deducted money and subtracted water inventory into the local message table and put it into the same transaction (relying on database local transactions to ensure consistency).

two。 At this time, there is a scheduled task to poll the local transaction table, throw the unsent message to the commodity inventory server, and ask it to subtract the water inventory. After arriving at the commodity server, it has to write to the transaction table of this server first. Then the deduction is carried out. After the deduction is successful, update the status in the transaction table.

3. The commodity server scans the message table through a scheduled task or notifies the deduction server directly, and the deduction server updates the status in the local message table.

4. In view of some abnormal situations, the messages that were not processed successfully are scanned regularly and re-sent. After the commodity server receives the message, it is first judged whether it is duplicated or not.

If it has been received, then determine whether it will be executed, and if the execution will immediately notify the transaction; if not, you need to re-execute idempotent guaranteed by the service, that is, you will not deduct an extra bottle of water.

Local message queue is the BASE theory and the ultimate consistent model, which is suitable for situations where the requirement of consistency is not high. When implementing this model, you need to pay attention to the idempotent of retry.

MQ transaction

The distributed transaction is implemented in RocketMQ, which is actually an encapsulation of the local message table, moving the local message table to the inside of MQ.

Here's a brief introduction to MQ transactions, which you can refer to if you want to learn more about them: https://www.jianshu.com/p/453c6e7ff81c.

The basic process is as follows:

In the first stage of Prepared message, you will get the address of the message.

The second phase executes the local transaction.

The third stage accesses the message through the address obtained in the first stage and modifies the status. The recipient of the message can use the message.

If the confirmation message fails, a scheduled scan for messages with no update status is provided in the RocketMQ Broker.

If a message is not acknowledged, a message is sent to the sender to determine whether it is submitted or not. In RocketMQ, it is sent to the sender in the form of Listener for processing.

If the consumption times out, it needs to be retried all the time, and the message receiver needs to be idempotent. If the message consumption fails, it needs to be processed manually, because the probability is low, and the loss outweighs the gain if this complex process is designed for such a small probability of time.

Saga transaction

Saga is a concept mentioned in a database ethics article 30 years ago. Its core idea is to split the long transaction into several local short transactions, which is coordinated by the Saga transaction coordinator. If it ends normally, it is completed normally. If a step fails, the compensation operation is called once according to the opposite order.

Composition of Saga: each Saga consists of a series of sub-transaction Ti, and each Ti has a corresponding compensation action Ci, which is used to undo the results caused by Ti. Each T here is a local transaction.

As you can see, compared to TCC, Saga does not have a "reserved try" action, and its Ti is submitted directly to the library.

There are two orders in which Saga is executed:

T1,T2,T3,...,Tn .

T _ 1, T _ 2, and C _ 1, where 0 < j < n.

Saga defines two recovery strategies:

Reversing backward, the second execution order mentioned above, where j is the sub-transaction where the error occurred, has the effect of undoing all previous successful sub-transation, causing the execution result of the entire Saga to be undone.

Forward recovery, suitable for scenarios that must be successful, the order of execution is similar to this: T1MagneT2MagneTj (failed), Tj (retry),..., Tn, where j is the sub-transaction where the error occurred. Ci is not required in this case.

It is important to note that isolation is not guaranteed in Saga mode because other transactions can still override or affect the current transaction without locking resources.

Or take the example of buying a bottle of water for 100 yuan. Here's the definition:

T1 = deduct 100 yuan, T2 = add a bottle of water to the user, T3 = reduce the stock of a bottle of water.

C1 = add 100 yuan, C2 = subtract a bottle of water for the user, C3 = add a bottle of water to the inventory.

If there is a problem, we perform the reverse of the C operation in which the problem occurs.

The isolation problem mentioned above will occur if the rollback needs to be performed by T3, but the user has already drunk the water (another transaction) and will find that it is impossible to reduce a bottle of water to the user when rolling back.

This is the problem of no isolation between transactions. You can see that the impact of Saga mode without isolation is still greater. You can refer to Huawei's solution: from the business level, add a Session and lock mechanism to ensure serialization of operation resources.

You can also isolate these resources at the business level by freezing funds in advance, and finally get the latest updates by reading the current status in a timely manner during the business operation. (specific example: please refer to Huawei's Service Comb)

Last

Again, if you don't need distributed transactions, if you have to, combine your own business analysis to see which one your business is more suitable for, whether you care about strong consistency or final consistency.

Finally, in summing up some questions, you can come down and find the answer from the article:

Is the CA of ACID and CAP the same?

What are the advantages and disadvantages of common solutions for distributed transactions? What scenarios are suitable for?

What are the reasons for distributed transactions? To solve what pain points?

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.