Shulou(Shulou.com)06/03 Report--

This article introduces the relevant knowledge of "what is the urgent need of Kafka for Zookeeper". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

1. Classic usage scenarios of Zookeeper

Zookeeper is accompanied by big data and the rise of distributed fields. A very important issue in big data is how to use many cheap machines to achieve reliable storage.

The so-called cheap machine means that the probability of failure is very high, but the cost of a single machine is also very low. The distributed domain wants to use multiple machines to form a cluster and store data on multiple machines (copies). In order to facilitate data consistency, it is usually necessary to select a master node from a replication group to handle data reading and writing, and other nodes copy data from the master node when the master node goes down. Automatic re-election is required to achieve high availability.

In the above scenario, there is a very important function of Leader election, how to elect a master node and support the master node to automatically trigger re-election when the master node goes down, to achieve automatic master-slave switching and high availability.

Using the temporary sequential node and event listening mechanism provided by Zookeeper, the Leader election can be easily realized.

The above T1 T2 can be understood as multiple members of an organization who can provide the same service, but in order to achieve the cold standby effect (that is, only one member provides services at the same time, we call it Leader. When Leader goes down or stops service, other famous members of the organization re-compete for Leader and then continue to provide services).

As shown in the figure above, Zookeeper is deployed as a cluster, which can effectively avoid a single point of failure, and provides strong consistency of data within the cluster.

When members need to compete for Leader, the implementation with Zookeeper is to create two children to a data node (/ app/order-service/leader in the example) node in zookeeper, which is a sequential temporary node.

The client determines whether the sequence number of the created node is the node with the lowest sequence number in / app/order-service/leader, and if so, it becomes Leader to provide services.

If the serial number is not the smallest, delete the event to the registered node in front of it. Once the process represented by Leader goes down and its session with Zookeeper expires, the temporary node associated with it will be deleted. Once the node created by Leader is deleted, the subsequent node will be notified, which triggers the elector again, elects a new Leader, continues to provide external services, and ensures the high availability of quality services.

Reviewing the above scenarios, it is very easy to select users with the help of Zookeeper, which makes it easy to improve the availability of applications, mainly by making use of several features of Zookeeper:

Temporary node

A temporary node is associated with a session. The session that created the temporary node ends at 1: 00, and it will be automatically deleted without the need for the application to delete it manually.

Sequential node

Event mechanism

With the help of the event mechanism, Zookeeper can timely notify other surviving application nodes and re-trigger the election, which makes it very easy to realize automatic master-slave switching.

2. Kafka's urgent need for Zookeeper.

There are many Leader elections in Kafka. Friends who are familiar with Kafka should know that a topic can have multiple partitions (data shards), and each data shard can be configured with multiple replicas. How to ensure the consistency of a partition's data among multiple replicas has become an urgent need.

The implementation routine of Kafka is multiple copies of a partition, from which a Leader is selected to undertake the read and write requests of the client, the slave node copies the content from the master node, and the Leader node decides whether the write is successful according to the success of the data written in the replica.

Schematic diagram of the partition distribution of topic in Kafka:

Therefore, the Leader election is needed here, and based on the fact that Zookeeper can easily achieve it, it has hit it off and started a "honeymoon".

3. Achilles' heel of Zookeeper

Zookeeper is a cluster deployment, which can provide services as long as more than half of the nodes in the cluster are alive. For example, a Zookeeper with 3 nodes allows 1 Zookeeper node to go down, but the cluster can still provide services; a Zookeeper with 5 nodes allows 2 nodes to go down.

But Zookeeper is designed as a CP model, that is, to ensure strong consistency of data, it is necessary to make sacrifices in terms of usability.

There are also so-called Leader nodes and slave nodes in the Zookeeper cluster. The Leader node is responsible for writing, and the Leader and slave nodes are available to accept read requests, but in the Zookeeper internal nodes, the whole Zookeeper can not provide services during the election. Of course, under normal circumstances, the election will be very fast, but in abnormal circumstances, it is difficult to say, such as the occurrence of full Gc on Zookeeper nodes, the impact will be devastating.

If Full Gc occurs frequently on a Zookeeper node, the session with the client will time out, and the temporary node associated with the session will be deleted because it is unable to respond to the client's heartbeat request (Stop World). Note that at this time, all temporary nodes will be deleted, the event notification mechanism that Zookeeper depends on will fail, and the election service of the entire cluster will become invalid.

From the perspective of high availability, the availability of Kafka clusters depends not only on itself, but also on external components, which is obviously not an elegant solution in the long run.

With the continuous improvement of related technologies in the distributed field, the idea of decentralization is gradually rising, and the voice of de-Zookeeper is getting louder and louder. In this process, a very excellent algorithm: Raft protocol has emerged.

Two important components of Raft protocol: Leader election and log replication, while log replication provides strong consistency for multiple replicas to provide strong data consistency, and a remarkable feature is that the Raft node is a decentralized architecture that does not rely on external components, but is embedded in the application as a protocol suite, that is, it is integrated with the application itself.

Then take the distribution map of Kafka Topic as an example, and the example diagram citing Raft protocol is as follows:

This is the end of the content of "what is the urgent need of Kafka for Zookeeper". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.