Shulou(Shulou.com)06/01 Report--

This article will explain in detail what is the core principle and application scenario of ZooKeeper. The content of the article is of high quality, so the editor will share it with you for reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

Why is there ZooKeeper?

We know that it is very difficult to write a distributed application, mainly because of local failures. When a message is transmitted between two nodes through the network, if the network fails, the sender does not know whether the receiver has received the message. There may be a network failure after receiving the message, it may be that the message has not been received, or the receiver's process may be dead. The only way for the sender to confirm is to connect again and send a message and ask him. This is a local failure: you don't know if the operation failed at all. Therefore, most distributed applications need a master and coordination controller to manage physically distributed child processes. Therefore, most applications need to develop private coordinator programs, which is a waste of time. At this time, a general-purpose and scalable coordinator is needed. It is because of such a scenario that ZooKeeper arises at the historic moment, and ZooKeeper is designed to reduce the coordination tasks undertaken by distributed applications.

Common application scenarios of ZooKeeper

01 / distributed coordination

To put it simply, distributed coordination means that someone listens to the data in the ZooKeeper. If the monitored data in the ZooKeeper is modified, the ZooKeeper will in turn tell the initiator of the change in the data. For example, in the design of Kafka, a node of Kafka creates a data in ZooKeeper. The strategy of Kafka is that whoever creates the data is the master node of the Kafka cluster, and the rest of the nodes will listen to the data. If the master node goes down, the data corresponding to the ZooKeeper will be changed, and the rest of the nodes listening to the data will perceive the master node downtime and hold a new election.

02 / metadata Management

Many distributed programs need to centrally manage their own metadata, and ZooKeeper is a good choice at this time. Distributed tools such as Kafka,Storm store the core metadata in the cluster in ZooKeeper.

03 / High availability

Many distributed projects are master-slave architectures. normally, one of the cluster nodes is the master node, and the rest are slave nodes. However, if there is only one master node, the program will have a single point of failure, so it is necessary to deploy multiple master nodes to achieve high availability, using ZooKeeper to select one of the multiple master nodes as master and the rest as StandBy. For example, the famous HDFS is highly available through ZooKeeper.

04 / distributed lock

Many projects in the enterprise need distributed locks. We can use ZooKeeper to do distributed locks, but we should note here that ZooKeeper can indeed do distributed locks, but ZooKeeper does not support too high concurrency, that is to say, if it is high concurrency, ZooKeeper may not be suitable for distributed locks. If you use Redis to do distributed locks in high concurrency cases, it is recommended that you use Redis to do distributed locks. But when the concurrency is not too high, it is convenient to use ZooKeeper to do distributed locks, and many people do use them.

Core principle of ZooKeeper

01 / ZooKeeper cluster architecture

In a ZooKeeper cluster, the server roles in the cluster are divided into leader and learner,learner, and then observer and follower. The specific functions are as follows:

0x01, leader (Leader)

Provide read and write functions for the client, responsible for the initiation and decision of voting. Only leader in the cluster can accept the write service.

0x02, follower (follower)

Provides a read service for the client and, if it is a write service, forwards it to leader. To vote during the election process.

0x03, observer (Observer)

Provide a read service for the client, and forward a leader if it is a write service. Do not vote in the leader election. Nor participate in the mechanism of more than half of the principles of writing. Improve the performance of cluster reads without affecting writing, which is a new role in the zookeeper3.3 series.

0x04, client (client)

The user who connects to the zookeeper cluster, the initiator of the request, is independent of the role of the zookeeper cluster.

02 / ZooKeeper read-write mechanism

In the ZooKeeper election, if more than half of the nodes choose a node as leader, then this node will be a leader node. For this reason, in a ZooKeeper cluster, as long as more than half of the nodes are alive, the ZooKeeper can provide services normally. For example, there are 5 ZooKeeper nodes, of which 2 nodes are down. At this time, 3 nodes are still alive, and the number of surviving nodes is more than half. At this time, the cluster still provides services normally, so there is no high availability problem with ZooKeeper cluster. And because the judgment of survival is based on more than half, we usually use odd numbers when building ZooKeeper clusters, which will save machines. For example, if we install a 6 ZooKeeper cluster, if we have 3 ZooKeeper clusters down, the cluster will become unavailable. Because there are no more than half of the nodes alive at this time, the effect of 6 is the same as that of 5. It is more appropriate for us to use 5.

Corresponding to a ZooKeeper cluster, we may have multiple clients, and the client can connect to one of the ZooKeeper nodes arbitrarily, but all clients can only write data to the leader node, and all clients can read data from all the nodes. If a client connects to a follower node and then sends a write request to the follower, follower forwards the write request to the leader node for processing. When leader receives a write request, it will synchronize data to other nodes (including itself). If more than half of the nodes receive the message and send back the ack message, then the leader node will commit,commit the message and the message will be visible to the user. Since leader does not commit messages until more than half of the nodes send ack, there will be a problem at this time. If the cluster is larger, the longer it takes to wait for more than half of the nodes to send back ack messages, that is, the more nodes will increase the read performance of the cluster, but it will affect the write performance of the cluster. Therefore, we generally recommend that the cluster size of ZooKeeper is about 3 to 5 nodes. To solve this problem, an observer role was added to the later ZooKeeper, which did not vote, but was only responsible for synchronizing data. For example, when we lead to write data, more than half of the nodes need to send ack responses, and this observer node does not participate in more than half of the statistics. It is only responsible for synchronizing data from leader and then providing it to the client for reading, so the purpose of introducing this role is to increase the performance of cluster reads without affecting the write performance of the cluster. Users can set up this role when setting up a cluster.

03 / Zookeeper characteristics

0x01, consistency

Client clients read the same data no matter which node in the cluster they connect to.

0x02, real-time

ZooKeeper ensures that clients get results within a certain time interval, including success and failure, but due to network delay, ZooKeeper cannot guarantee that both clients get newly updated messages at the same time. If you need the latest messages, you need to call the sync () interface.

0x03, atomicity

When leader synchronizes data, the synchronization process ensures that it is transactional and either succeeds or fails.

0x04, sequentiality

If message an is published before message b on a server, then all messages an on server are published before message b.

04 / Zookeeper data consistency guarantee

We just saw that ZooKeeper has a number of features, but I believe that among them, people are most curious about how Zookeeper ensures data consistency. ZooKeeper uses the ZAB protocol to ensure data consistency. Data synchronization between ZooKeeper clusters is carried out through this protocol to ensure data consistency.

0x01, two-phase commit + more than half write mechanism

The mechanism of ZooKeeper writing data is that the client sends the write request to the leader node (if the follower node is sent, the follower node will forward the write request to the leader node), and the leader node will send the data to all nodes (including itself) through the proposal request. After receiving the data, all the nodes will write to their own local disk, and after writing, they will send an ack request to leader. As long as leader receives an ack response from more than half of the nodes, it will send a commit message to each node, and each node will put the message in memory (to ensure high performance), and the message will be visible to the user. At this time, if ZooKeeper wants to ensure data consistency, you need to consider the following two situations: case 1: leader has executed commit, and leader is down before sending commit to follower. How to ensure message consistency at this time? Case 2: the client has written the message to leader, but the leader has not sent the proposal message to other nodes. At this time, the leader is down. What should I do when the leader is restored after the downtime?

Crash recovery Mechanism of 0x02 and ZAB

In case 1, when the leader goes down, the ZooKeeper will elect a new leader. After the new leader starts, it will go to the disk to check whether there is a message without commit. If so, continue to check whether other follower have commit the message. If more than half of the nodes have ack the message but no commit, then the new leader needs to complete the commit operation.

Delete data mechanism in 0x03 and ZAB recovery

For case 2, the client has written the message to leader, but leader has not sent the portal message to other nodes. At this time, leader is down. For the user, the message failed to write. Suppose that after a period of time, the leader node resumes again, but at this time the role becomes follower. When it checks its disk, it will find that it has a message that has not been commit. At this time, it will detect the number of the message. The message is numbered, which is composed of high 32 bits and low 32 bits. The high 32 bits are used to reflect whether leader switching has occurred, and the low 32 bits show the order of messages. At this time, the current node will know that the current leader has been switched according to the high 32 bits, so delete the current message and then synchronize the data from the new leader, thus ensuring data consistency.

So much for sharing about the core principles and application scenarios of ZooKeeper. I hope the above content can be helpful to you and learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.