Shulou(Shulou.com)05/31 Report--

Open source distributed transaction solution Fescar full parsing is like, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain in detail for you, people with this need can come to learn, I hope you can get something.

The well-known Ali distributed transaction solution: GTS (Global Transaction Service), has officially launched an open source version, named "Fescar", hoping to help the industry solve the distributed transaction problem under the micro-service architecture. Today, let's have an in-depth understanding.

Microservices advocate splitting complex single applications into several services with simple functions and loose coupling, which can reduce the difficulty of development, enhance scalability and facilitate agile development. At present, it is respected by more and more developers. After the micro-service of the system, a seemingly simple function may need to invoke multiple services and operate multiple database implementations, and the distributed transaction problem of service invocation becomes very prominent. Distributed transaction has become the biggest obstacle to the landing of micro-services, and it is also one of the most challenging technical problems.

1. What is the distributed transaction problem caused by microservice?

First of all, imagine a traditional monolithic application (Monolithic App), through three Module, update data on the same data source to complete a business.

Naturally, the data consistency of the entire business process is guaranteed by local transactions.

Cdn.com/b739529e7894d04f964871a75529b427d4fcd879.png ">

With the change of business requirements and architecture, single applications are split into micro-services: the original three Module are split into three independent services, using independent data sources (Pattern: Database per service). The business process will be completed by calls to three services.

At this point, the data consistency within each service is still guaranteed by local transactions. And how to ensure the global data consistency at the whole business level? This is the typical distributed transaction requirement under the micro-service architecture: we need a distributed transaction solution to ensure the global data consistency of the business.

2. The development of Fescar

Ali is the first batch of domestic enterprises to carry out application distributed (micro-service) transformation, so it has encountered the problem of distributed transaction under micro-service architecture for a long time.

In 2014, the Ali middleware team released TXC (Taobao Transaction Constructor) to provide distributed transaction services for intra-group applications.

In 2016, after product transformation, TXC landed in Aliyun as GTS (Global Transaction Service), becoming the only cloud-based distributed transaction product in the industry at that time, and began to serve many external customers in Ayunli's public and private cloud solutions.

Since 2019, based on the technology accumulation of TXC and GTS, the Ali middleware team launched the open source project Fescar (Fast & EaSy Commit And Rollback, FESCAR) to build this distributed transaction solution with the community.

In a continuous line, TXC/GTS/Fescar has provided a unique answer to solve the problem of distributed transactions under the micro-service architecture.

2.1 original intention of the design

In the fast-growing Internet age, the ability to try and make mistakes quickly is critical to the business:

On the one hand, it should not bring additional R & D burden to the business level because of the micro-service of technical architecture and the introduction of distributed transaction support.

On the other hand, the services supported by distributed transactions should basically maintain the performance of the same order of magnitude, and the transaction mechanism should not significantly slow down the business.

Based on these two points, the most important consideration at the beginning of our design is:

No intrusion into the business: "intrusion" here means that the application is required to be designed and modified at the business level because of the technical constraints of distributed transactions. This kind of design and transformation often brings high R & D and maintenance costs to the application. We want to solve the distributed transaction problem at the middleware level without requiring the application to do extra work at the business level.

High performance: the introduction of distributed transaction guarantee will inevitably lead to additional overhead, resulting in performance degradation. We want to reduce the performance loss of distributed transactions to a very low level, so that applications will not be affected by the introduction of distributed transactions.

2.2 Why are existing solutions not satisfied?

The existing distributed transaction solutions are divided into two categories according to the intrusiveness to the business, namely, non-intrusive and intrusive to the business.

Non-intrusive solution for business

Among the existing mainstream distributed transaction solutions, only the XA-based solution does not invade the business, but there are three problems in the application of XA scheme:

Database is required to provide support for XA. If you encounter a database that does not support XA (or does not support it well, such as a version of MySQL prior to 5.7), you cannot use it.

Due to the constraints of the protocol itself, the locking period of transaction resources is long. From a business point of view, long-period resource locking is often unnecessary, and because the manager of the transaction resource is the database itself, the application layer cannot intervene. The resulting situation is that XA-based applications tend to have poor performance and are difficult to optimize.

The XA-based distributed solutions that have been implemented all rely on heavyweight application servers (Tuxedo/WebLogic/WebSphere, etc.), which are not suitable for micro-service architecture.

A scheme to invade the business

In fact, XA was the only solution for the initial distributed transaction. XA is complete, but in practice, due to various reasons (including but not limited to the three points mentioned above), we often have to give up and start from the business level to solve the problem of distributed transactions. For example:

Final consistency Scheme based on reliable message

TCC

Saga

All fall into this category. The specific mechanism of these programs is not carried out here, and there are many articles on this aspect on the Internet. In short, these schemes require the technical constraints of distributed transactions to be taken into account in the design at the business level of the application, and usually each service needs to design and implement forward and reverse idempotent interfaces. Such design constraints often lead to high R & D and maintenance costs.

2.3 what should the ideal plan look like?

It is undeniable that the distributed transaction solutions that invade the business have been verified by a large number of practices, can effectively solve the problem, and play an important role in the business application systems of various industries. But back to the origin, the adoption of these programs is actually forced by helplessness. Imagine that if the XA-based solution can be less heavy and can guarantee the performance requirements of the business, I believe that no one will be willing to solve the distributed transaction problem at the business level.

An ideal distributed transaction solution should be as simple as using local transactions, and the business logic only focuses on the requirements at the business level, without considering the constraints of the transaction mechanism.

3. Principle and design

We want to design a non-intrusive solution to the business, so consider from the business non-intrusive XA scheme: can we evolve on the basis of XA to solve the problems faced by the XA solution?

3.1 how do you define a distributed transaction?

First of all, naturally, we can think of a distributed transaction as a global transaction that contains several branch transactions. The responsibility of a global transaction is to coordinate the branch transactions under its jurisdiction to reach an agreement, either successfully commit together or fail to roll back together. In addition, usually the branch transaction itself is a local transaction that satisfies ACID. This is our basic understanding of the distributed transaction structure, which is consistent with XA.

Secondly, similar to XA's model, we define three components to agree on the processing of distributed transactions.

Transaction Coordinator (TC): transaction coordinator, which maintains the running state of global transactions and is responsible for coordinating and driving the commit or rollback of global transactions.

Transaction Manager (TM): controls the boundaries of a global transaction, is responsible for opening a global transaction, and ultimately initiates a global commit or global rollback resolution.

Resource Manager (RM): control branch transactions, responsible for branch registration, status reporting, and receive instructions from the transaction coordinator to drive the commit and rollback of branch (local) transactions.

A typical distributed transaction process:

TM requests TC to open a global transaction, and the global transaction is created successfully and a globally unique XID is generated.

XID propagates in the context of the microservice invocation link.

RM registers branch transactions with TC and puts them under the jurisdiction of XID corresponding global transactions.

TM initiates a global commit or rollback resolution against XID to TC.

TC schedules all branch transactions under the jurisdiction of XID to complete commit or rollback requests.

So far, the protocol mechanism of Fescar is generally consistent with that of XA.

3.2What is the difference with XA? Architecture level

The RM of the XA scenario is actually at the database layer, and RM is essentially the database itself (supplied for use by providing drivers that support XA).

The RM of Fescar is deployed on this side of the application as a middleware layer in the form of a two-party package, which does not depend on the support of the protocol with the database itself, and certainly does not need the database to support the XA protocol. This is very important for a micro-service architecture: the application layer does not need to adapt two different sets of database drivers for two different scenarios: local transactions and distributed transactions.

This design strips off the requirements of the distributed transaction scheme on the protocol support of the database.

Two-phase submission

Let's first take a look at the 2PC process of XA.

Regardless of whether the Phase2's decision is commit or rollback, the lock on the transactional resource will not be released until the Phase2 is complete.

Imagine a normal business, there is a high probability that more than 90% of the transactions should eventually be successfully committed, can we commit local transactions in Phase1? In more than 90% of the cases, the time of holding locks by Phase2 can be saved and the overall efficiency can be improved.

This design reduces the transaction lock time in most scenarios, thus improving the concurrency of transactions.

Of course, you must ask: how does Phase2 roll back when Phase1 is submitted?

3.3 how are branch transactions committed and rolled back?

First of all, the application needs to use Fescar's JDBC data source agent, that is, Fescar's RM.

Phase1:

Through parsing the business SQL, the JDBC data source agent of Fescar organizes the data mirroring of the business data before and after the update into a rollback log, and makes use of the ACID feature of the local transaction to submit the update of the business data and the writing of the rollback log in the same local transaction.

In this way, it is guaranteed that there must be a corresponding rollback log for the update of any submitted business data.

Based on this mechanism, the local transaction of the branch can be committed in the Phase1 of the global transaction and immediately release the resources locked by the local transaction.

Phase2:

If the resolution is a global commit, the branch transaction has completed the commit at this time, and there is no need for synchronous coordination processing (only the rollback log needs to be cleaned asynchronously), and Phase2 can complete very quickly.

If the resolution is a global rollback, RM receives a rollback request from the coordinator, finds the corresponding rollback log record through XID and Branch ID, and generates a reverse update SQL through the rollback record and executes to complete the branch rollback.

3.4 transaction propagation mechanism

XID is the unique identification of a global transaction. What the transaction propagation mechanism needs to do is to pass the XID in the service invocation link and bind it to the transaction context of the service. In this way, the database update operations in the service link will register branches with the global transaction represented by the XID and fall under the jurisdiction of the same global transaction.

Based on this mechanism, Fescar can support any micro-service RPC framework. Just find a mechanism in a specific framework that transparently propagates XID, such as Dubbo's Filter + RpcContext.

Corresponding to the transaction propagation attributes defined by the Java EE specification and Spring, Fescar supports the following:

PROPAGATION_REQUIRED: supported by default

PROPAGATION_SUPPORTS: supported by default

PROPAGATION_MANDATORY: applications are implemented through API

PROPAGATION_REQUIRES_NEW: applications are implemented through API

PROPAGATION_NOT_SUPPORTED: applications are implemented through API

PROPAGATION_NEVER: applications are implemented through API

PROPAGATION_REQUIRED_NESTED: not supported

3.5 isolation

The isolation of global transactions is based on the local isolation level of branch transactions.

On the premise that the local isolation level of the database has been committed or above, Fescar designs a global write exclusive lock maintained by the transaction coordinator to ensure write isolation between transactions, and defines the global transaction at the read uncommitted isolation level by default.

Our consensus on isolation levels is that most applications have no problem reading submitted isolation levels. In fact, there are most of these application scenarios, and in fact, there is no problem working under the isolation level of reading uncommitted.

In extreme scenarios, if the application needs to achieve global read submission, Fescar also provides a corresponding mechanism to achieve the goal. By default, Fescar works at the isolation level of read and no commit, ensuring the efficiency of most scenarios.

The embodiment of the ACID attribute of a transaction in Fescar is a complex topic. We will have a special article to analyze it in depth, and we will not expand it further here.

4. Applicable scenario analysis

There is an important premise in the core principle of Fescar mentioned earlier: the resources involved in branch transactions must be relational databases that support ACID transactions. The commit and rollback mechanisms of branches depend on the guarantee of local transactions. Therefore, it is not applicable if the application uses a database that does not support transactions, or is not a relational database at all.

In addition, there are still some limitations in the current implementation of Fescar, such as: the transaction isolation level is up to the level of read committed, and the parsing of SQL can not cover all syntax.

In order to override the fact that the native mechanism of Fescar cannot support application scenarios, we define another working mode.

The Fescar native working mode described above is called AT (Automatic Transaction) mode, which is non-intrusive to the business. Another corresponding working mode is called MT (Manual Transaction) mode, in which branch transactions need to apply themselves to define the business itself and the logic of commit and rollback.

4.1 basic behavior patterns of branches

Branch transactions, which are part of a global transaction, contain four behaviors that interact with the coordinator, except for their own business logic:

Branch registration: before the data operation of the branch transaction is carried out, it is necessary to register with the coordinator to bring the upcoming branch transaction data operation into the management of an open global transaction. Only after the branch registration is successful.

Status escalation: after the data operation of the branch transaction is completed, the execution result needs to be reported to the transaction coordinator.

Branch commit: the branch commit is completed in response to the request made by the coordinator for the branch transaction commit.

Branch rollback: completes the branch rollback in response to a request from the coordinator for a branch transaction rollback.

4.2 behavior patterns of AT pattern branches

The business logic does not need to pay attention to the transaction mechanism, and the interaction between the branch and the global transaction occurs automatically.

4.3 behavior patterns of MT pattern branches

The business logic needs to be broken down into Prepare/Commit/Rollback 3 parts to form a MT branch to join the global transaction.

On the one hand, MT mode is a supplement to AT mode. In addition, a more important value is that many non-transactional resources can be brought into the management of global transactions through the MT model.

4.4 mixed mode

Because the branches of the AT and MT patterns are fundamentally consistent in behavior, they are fully compatible, that is, branches of both AT and MT can exist in a global transaction. In this way, we can achieve the goal of fully covering the business scenario: AT mode can support, use AT mode; AT mode can not support temporarily, use MT mode instead. In addition, naturally, non-transactional resources managed by MT mode can also be incorporated into the management of the same distributed transaction together with relational database resources that support transactions.

4.5 Vision of the application scenario

Back to the original intention of our design: an ideal distributed transaction solution should not invade the business. MT mode is a natural supplement when AT mode can not completely cover all scenes for the time being. We hope that through the continuous evolution and enhancement of the AT pattern, the supported scenarios will be gradually expanded and the MT pattern will gradually converge. In the future, we will include native support for XA and use XA as a non-invasive way to cover scenarios that cannot be reached by AT mode.

5. Extension point 5.1 support for the micro-service framework

The propagation of transaction context between microservices needs to customize the optimal solution transparent to the application layer according to the mechanism of the microservice framework itself. Developers interested in co-building in this area can refer to the built-in support for Dubbo to support other micro-service frameworks.

5.2 supported database types

Because AT involves parsing SQL, there are specific adaptations for working on different types of databases. Developers who are interested in co-building in this area can refer to the built-in support for MySQL to support other databases.

5.3 configuration and service registration discovery

Support access to different configurations and service registration discovery solutions. For example: Nacos, Eureka, ZooKeeper and so on.

Scene extension of 5.4 MT mode

One of the important functions of MT schema is that it can bring the resources of non-relational database into the jurisdiction of global transaction through the packaging of MT schema branch. For example, transaction messages for Redis, HBase, RocketMQ, and so on. Developers interested in co-building in this area can contribute a series of ecological adaptation solutions here.

5.5 distributed high availability scheme for transaction coordinator

For different scenarios, different ways are supported as highly available solutions on the Server side of the transaction coordinator. For example, the persistence of transaction state can be based on file or database, and the state synchronization between clusters can be based on RPC communication or highly available KV storage.

6. Roadmap blueprint

The green part has been released by open source, the yellow part will be released by Ali in subsequent versions, and the blue part is the ecological part that we build with the community:

For support for different databases, developers can refer to the implementation of MySQL.

For support for different micro-service frameworks, developers can refer to the implementation of Dubbo.

For support for MQ and NoSQL, developers can refer to the implementation of TCC.

Configuration and service registration discovery: developers can access any framework that can provide such services with a small amount of work.

Of course, the non-blue part is also very welcome for the community to participate and contribute better solutions.

In addition, as the standard of distributed transaction, XA is indispensable to a complete distributed transaction solution. In the long-term planning, we must add the support of XA.

Preliminary version planning v0.1.0:

Microservice framework support: Dubbo

Database support: MySQL

Annotation based on Spring AOP

Transaction coordinator: stand-alone version

V0.5.x:

Microservice framework support: Spring Cloud

MT mode

Support the adaptation of TCC mode transactions

Dynamic configuration and service discovery

Transaction coordinator: highly available cluster version

V0.8.x:

Metrics

Console: monitor / deploy / upgrade / scale up

V1.0.0:

General Availability: suitable for production environment

V1.5.x:

Database support: Oracle/PostgreSQL/OceanBase

Annotation that does not depend on Spring AOP

Optimal processing Mechanism of Hot spot data

RocketMQ transaction messages are integrated into global transaction management

Adaptation Mechanism for integrating NoSQL into Global transaction Management

Support for HBase

Support for Redis

V2.0.0:

Support for XA

Of course, in the process of iterative evolution of the project, we pay most attention to the voice of the community, and the roadmap will be fully communicated with the community and adjusted in a timely manner.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.