Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "how to solve the common distributed transaction theory of java". In the operation of practical cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

CAP theory

Start with the definition:

C (Consistence): consistency

What does it mean that all nodes access the latest copies of the data? We know that in a distributed system, in order to be highly available, a node often has several copies of data, referred to as Follower nodes. The more common pattern is that our data updates are generally written to Leader nodes, and then synchronized to Follower nodes. When we read data, we can read the latest data no matter from which node, this is consistency.

A, B and C can get the same data.

A (Availability): availability

The non-failure node can operate normally. To put it simply, the client can always access and get the normal response of the system. From the user's point of view, there will be no problems such as system operation failure or access timeout. However, problems such as network delay may occur within the system.

Because the C node is unavailable, it will be eliminated normally, and there may be synchronization delay between B node and A node, but B node itself has no fault, so B node is available.

P (Partition tolerance): partition fault tolerance

The problem of the network is complex, and the distributed system must consider this point. If a node causes data inconsistency due to problems such as the network, or the data is synchronized for a long time, although it will affect the timeliness of some node data, the service node is still available, and the distributed system should be able to tolerate this situation.

Although the node corresponding to B has lost contact with Leader, it can still provide external services, but it only provides old data.

In distributed systems, CAP cannot be satisfied at the same time. First of all, because there are multiple nodes and network transmission takes time, there may be delays, so we can not guarantee that the data between nodes are completely consistent at a certain time, so P (partition fault tolerance) should be satisfied. When P is satisfied, why can't CA be satisfied at the same time? Let's take a hypothetical look at what happens if CA is satisfied at the same time.

Suppose that C (consistency) is required now, that is to say, all nodes must provide the same data at some point. We know that in the case of P, it is impossible to guarantee that we can only kill all other nodes. For example, if reading and writing are prohibited, this is actually contrary to A (although some nodes are delayed, but the nodes themselves are available).

Assuming that A (availability) is now required, that is, as long as there is nothing wrong with the node itself, it can provide services, even if there is a bit of data delay, which is obviously contrary to C.

In the actual business, we need to decide whether to use CP or AP according to the business scenario. For example, for some businesses linked to money, data consistency should be the most important in reason, so CP is generally used, while for some businesses that do not affect the main functions, such as news reading, different users will not see the same amount of reading will not cause any impact, you can use AP.

BASE theory

Because C and A can not be both in CAP theory, eBay architects put forward BASE theory, BASE theory is mainly between CA, it does not require strong consistency, so it can meet a certain degree of usability. Let's start with the definition:

BA (Basically Available): basic availability

Note that this is not the same thing as unavailability. When an unpredictable failure occurs in a distributed system, partial availability is allowed to be lost to ensure the availability of core functions. For example, a normal interface responds to 200ms and responds for more than 1 second when a failure occurs. Although the response time is longer, the interface can still provide services. For example, for a video website, when a burst of traffic arrives. Hang up the video on-screen comment service, but the video playback function is still normal.

S (Soft-state): soft statu

That is, the distributed system allows the existence of an intermediate state, but this intermediate state does not have a serious impact on the service, such as for master-slave replication, which allows a short delay from the slave node.

E (Eventually Consistent): final consistency

Due to the existence of the soft state, the system can tolerate delay, but after a period of time, the delayed data needs to be finally consistent.

Generally speaking, the applicability of BASE theory should be more extensive. In many cases, we do not require strong consistency of data, as long as we can achieve consistency after a short delay.

Consistent hash

The word hash is not unfamiliar to us. As for cache servers, several servers are generally configured online, and then which cache service is requested according to hash. For example, the common way is to obtain the target machine by using hash (key)% num.

Suppose there are three cache servers, and there are currently three cached key, namely K0Magol K1PowerK2. After hash, the distribution of them is as follows:

Hash (K0)% 3 # No.0hash (K1)% 3% 1 # No.1hash (K2)% 3% 2 # No.2

Fortunately, it is very evenly distributed, one for each machine. One day, due to an online activity, the number of visits is expected to increase, and we need to choose an additional server to share the pressure, so after hash, the distribution of k0memk1memk2 is as follows:

Hash (K0)% 4 # No.1hash (K1)% 4 # No.2hash (K2)% 4 # No.3

The target cache server of K0 has been changed from No.0 to No.1.

The target cache server of K1 has changed from No.1 to No.2.

The target cache server of K2 has changed from No.2 to No.3.

It can be found that due to the addition of a cache node, all the original caches of K0PersonK1PowerK2 have become invalid, which seems to be a bit of a problem, similar to the cache avalanche, which will cause great pressure on DB if it is serious. The main reason for this problem is that we have added a node, resulting in changes in hash results, and hash can be said to be unstable at this time.

In order to solve the problem of rehash instability, the consistent hash algorithm appears. The principle of consistent hash is relatively simple. First, there is a hash ring that can hold 0-2 ^ 32-1 nodes.

The first step is to map our target server node to this ring through hash

In the second step, according to the key we need to find, it should also correspond to a location on the hash ring.

You may ask that K0, K1, and K2 are not aligned with a cache node. This is where consistency hash is different. At this time, it does not look for hash (key) = a node, but finds the first node clockwise according to the location of the key. This node is the target node of the current key.

Let's take a look at what happens when a new node is added in the case of consistent hash.

The only change at this time is that K0 should have hit the cache0 node, but now it has hit our newly added node cache3, while K1 and K2 remain unchanged, that is, the cache with and only K0 is invalidated, which greatly reduces the area of cache invalidation compared to before.

Of course, such a node distribution is ideal, if our nodes are distributed like this:

Due to the clockwise search method, the distribution of several cache nodes is relatively concentrated, so in the end, K0Magol K1PowerK2 falls on the cache0 node, that is to say, cache1 and cache2 are basically superfluous, so in order to solve the problem of data skew, consistent hash introduces the concept of virtual node, each node can have several virtual nodes, such as:

Cache0- > cache0#1

Cache1- > cache1#1

Cache2- > cache2#1

Virtual node is not a real service node, it is just a shadow, its purpose is to stand pit, so that the nodes are more distributed and more uniform.

In this way, after mapping the virtual nodes, K0 to cache2,k1, to cache0,k2 to cache1, the more virtual nodes, the more uniform the theoretical distribution.

Gossip protocol

A cluster is often composed of multiple nodes. When a node joins the cluster or a node goes offline from the cluster, you need to let other nodes in the cluster know. Only in this way can you share the data information with the new node and ignore the offline node.

A, B and C nodes can transmit messages to each other, but D nodes will be broadcast to other surviving nodes after going offline.

This kind of broadcast protocol is to talk about Gossip protocol today, and Gossip protocol is also called Epidemic protocol (epidemic protocol). When a message arrives, it can infect all cluster nodes like a virus through Gossip protocol. Of course, we take advantage of its strong spreading ability.

The process of Gossip is initiated by a seed node. When a seed node has information that needs to be synchronized to other nodes in the network, it will randomly select several surrounding nodes to spread the message, and the node that receives the message will repeat the process until all the nodes in the network receive the message. This process may take a certain amount of time, so there is no guarantee that all nodes will have the message at a certain point in time, but in theory, all nodes will receive the message, so it is a final consistency protocol.

Characteristics of Gossip protocol:

The Gossip protocol spreads messages periodically and propagates at regular intervals.

Infected nodes can continue to spread N nodes at a time

Every time a message is spread, a node that has not yet been sent is selected for dissemination.

The node that receives the message will not spread it to the sending node

The same node may receive duplicate messages because multiple nodes may be spreading to it at the same time.

The cluster is decentralized and the nodes are equal.

The dissemination of the message does not have to wait for the ack of the receiving node, that is, the message may be lost, but it should eventually be infected.

Let's look at an example:

The seed node is A.

A node selects B and C nodes to spread

When C spreads to DMaget B to spread D and E, it can be found that D is received twice.

D spreads to F, and eventually the whole network synchronizes to the message.

Gossip is somewhat similar to the breadth-first traversal algorithm of graphs, which is generally used for the sharing and maintenance of network topology information, such as redis and consul.

Raft algorithm

One of the difficulties of distributed protocols is data consistency. When only one node in a cluster consisting of multiple nodes receives data, even if we are successful, the risk is too great. When all nodes are required to receive data, the response is successful, and the performance is too poor, so we generally make a compromise between data security and performance. As long as we ensure that most of the nodes synchronize data successfully, we will be successful. As a well-known consistency algorithm, Raft algorithm is widely used in many middleware, such as etcd. Let's take a look at how Raft algorithm works.

First of all, in the Raft algorithm, the corresponding roles of each node in several cases are introduced:

Leader node: like most distributed Leader nodes, data changes are made through it

Follower node: a follower of the Leader node, the node responsible for copying data and voting at election time

Candidate candidate node: the node participating in the election, which is the role that the Follower node will switch when participating in the election.

The Raft algorithm divides the consistency problem into two sub-problems, Leader election and state replication.

Election

First of all, let's take a look at the election of Leader. At the beginning of the system, all nodes are Follower nodes, and then everyone has the opportunity to participate in the election, that is, to turn themselves into Candidate, but if every Follower node becomes Candidate, then it will fall into an infinite endless loop, so each Follower has a timer, and the timer time is random, when the timer time of a certain Follower runs out. It will confirm whether the Leader node currently exists, and if it does not exist, it will turn itself into a Candidate. At this time, it will vote for itself and tell other nodes to vote. When more than half of the votes are obtained, the current Candidate will become a Leader node.

Because A node has the shortest timer time (10ms), A will become Candidate.

A votes for himself, while B and C also vote for their own consent, so A becomes the Leader node and records the M term. This M is for version checking, for example, a node with the number 10 receives a vote request from the node with the number 9, then the request will be rejected.

After the Leader node is elected, the Leader node will continue to send the heartbeat to other Follower nodes to prove that they are alive, and the other Follower nodes will clear their timers after receiving the heartbeat and reply to Leader, because there is no need to trigger the election at this time.

If the Leader node dies at some point, then the Follower node will not receive a heartbeat, so a new round of election will be triggered when the timer arrives, and the process is the same, but if both Follower happen to become Candidate and both get the same number of votes, then there will be a stalemate. In order to break the deadlock, each Candidate will postpone the request for a vote again for a period of time, of course, in general. The Candidate theory of first-come-first-served timer is more likely to become Leader.

The good election process is roughly the same as above, so let's take a look at data replication.

Copy

When the Leader node receives a change from Client, it records the change in log, and then Leader notifies the Follower node of the change with the next heartbeat, and the Follower node that receives the message also writes the change to the log, and then replies to the Leader node. When Leader receives most of the responses, it writes the changes to its own storage space, replying to client, and telling Follower to apply the log. At this point, the cluster reached a consensus on the change.

Finally, Raft algorithm is an algorithm that can achieve strong consistency in distributed systems. Each system node has three states: Leader, Follower and Candidate. The two most important things to implement Raft algorithm are: the election of the master and the replication of logs.

Distributed transaction

I believe that the nature of affairs is either to rush forward together or to remain motionless together. For MySQL's InnoDB, we just need to execute begin and commit, and sometimes we may need to roll back rollback. But this is under the premise of the same database, what if our data tables are divided into libraries, or what if the resources we want to operate are on different network nodes? This requires the distributed transactions we are going to talk about today, such as 2PC, 3PC, TCC, etc., but none of them can guarantee a perfect ACID. Let's take a look at what's going on.

2PC

As can be seen from the name, it is divided into two phases, so it is also called two-phase commit, that is, preparation and commit. 2PC requires a transaction coordinator. Compared with regular transactions, our request is sent to this coordinator, and then the coordinator helps us coordinate the submission of each node resource.

Preparation phase: the coordinator will ask the participants to do everything except commit, that is, waiting for the submission.

Submission phase: after receiving the preparation message from each participant, the coordinator notifies each participant to submit (commit) or roll back (rollback) according to the preparation.

You can find that the whole process is very dependent on the coordinator, and if the coordinator dies, then the entire distributed transaction is not available, so it is generally recommended that the coordinator have at least one backup node.

If, after receiving the ok of all nodes, when the coordinator is ready to send commit messages, one of the nodes will never receive the message due to network problems, then the node that does not receive the message will not release the resources all the time. When this happens, it is recommended that the coordinator have a retry function. After the commit fails, it keeps retrying until it succeeds. 2PC protocol is a strong consistency protocol, it is synchronous blocking, so its performance may have problems in high concurrency scenarios.

3PC

There are some problems in 2PC, such as the coordinator does not know the status of the current node from hanging to recovery, what to do now (whether to commit or roll back, etc.), and when network problems occur, the nodes that cannot communicate will only wait foolishly, resulting in resources being locked all the time. In view of these problems, 3PC has emerged.

First of all, as the name implies, 3PC will be divided into three phases, namely, the preparation phase, the pre-commit phase and the commit phase.

Preparation phase: the main task is to ask the participants about their own conditions, such as how your load is? Can you participate in the next mission?

Pre-submission phase: all the preparatory work except commit is waiting for commit

Commit phase: execute a real commit or rollback

If a new coordinator takes over during the transaction, it can determine what to do based on the status of a participant, for example, if a participant is in the commit phase, it indicates that the current transaction is in the commit phase. When a node cannot receive the submission information because of a network problem, it will not wait foolishly at this time, there will be a timeout, and when the timeout has passed, the node can submit it automatically, but here is a question: for the participant node, should it be commit or rollback?

In fact, neither 2PC nor 3PC can guarantee absolute consistency, because a participant node may not receive a message because of network problems, but other participant nodes have already submitted transactions. Generally, in order to prevent this problem, it is best to add an alarm, such as monitoring the information of differences automatically through scripts when a transaction exception is detected.

TCC

The whole process of TCC transaction is Try, Commit, Cancel,TCC transaction usage scenario is closer to the practical application, so it is also more widely used.

The Try:Try process generally refers to the process of locking resources, such as the common process of placing orders. In the try phase, we do not really reduce inventory, but lock the inventory that issued the order.

Commit: the real business logic is executed, with submission.

Cancel: undo. If Commit fails, the locked resource can be released back.

TCC is highly intrusive to applications. Each branch of business logic needs to implement three operations: try, confirm and cancel, and the cost of code modification is high. When a network or other system failure occurs, TCC needs to implement the corresponding rollback logic according to the actual business scenario. Commit or Cancel may try again, so the corresponding part had better support idempotent.

Finally, in theory, none of the above three kinds of distributed transactions can guarantee absolute consistency, because they cannot solve the unexpected factors caused by the network, so you can only retry indefinitely to solve them. However, this infinite retry is best to use message queuing + daemon to automatically supplement data, as long as the message queue does not lose data. In short, not only distributed transactions will bring these problems, distributed itself will also bring a lot of problems, there is no absolute solution, only a better solution.

This is the end of the content of "how to solve the common distributed transaction theory of java". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.