Shulou(Shulou.com)05/31 Report--

This article introduces how to analyze the accelerated removal of Zookeeper in Apache Kafka 2.7.0. The content is very detailed. Interested friends can use it for reference. I hope it will be helpful to you.

The Apache Kafka community, a distributed publish and subscribe system, released the belated version 2.7.0. Several key updates to this version include the continuous replacement of ZooKeeper in Apache Kafka, the addition of a new internal agent API, and the addition of a new Core Raft consensus algorithm. Now Apache Kafka has a Core Raft module that contains core consensus agreements alone. In addition, tiered storage is ongoing to provide unlimited expansion and faster rebalancing.

Zookeeper originally acts as a coordinating agent in Apache Kafka. When all proxy servers are started, they will connect to Zookeeper to register. When the state of the agent changes, Zookeeper will store this data, and the agent of Kafka will communicate with other agents through Zookeeper to synchronize, that is to say, Kafka cannot run smoothly without Zookeeper.

However, Zookeeper is not part of Kafka, so two systems must be deployed to run each Kafka cluster, which gives rise to many problems, including the consumption of excess resources, including more network, monitoring functions, security and other resource configuration, while the increase in the size of the Kafka cluster means that Zookeeper must expand and use more access, and Zookeeper as an external data storage service, when there is more and more data It makes the loading time of the controller longer and longer, which limits the expansion of the Kafka cluster.

So in 2019, the Apache Kafka community began to remove Zookeeper, with Kafka itself providing data management capabilities, while Apache Kafka 2.7.0 had a total of seven updates related to the removal of Zookeeper work, including KIP-497 's new internal agent API to replace the original internal synchronous copy (In-Sync Replica,ISR).

At present, the Kafka partition responsible program (Partition Leader) and ISR information are stored in Zookeeper, and both the controller and the partition responsible program can update this status, but because either party can update the status, there is a mechanism for sharing information, which will delay the update of ISR, which means that the data request may receive the old information.

Apache Kafka 2.7.0 adds a new AlterIsr API that gives the controller exclusive ability to update the status of the partition responsible program and ISR. The advantage of the new API is that data requests always get the latest status. It is officially mentioned that adding this API is an important step in removing ZooKeeper.

Because of the growing size of Kafka clusters, users need to store more data in Kafka, so they begin to introduce the concept of hierarchical storage. Kafka storage is now divided into local and remote tiers. Users can temporarily store data locally and then throw it to the remote end for longer-term storage. In this way, the time for local storage layer to retain data will be reduced from a few days to a few hours. Using the remote layer of a storage system such as HDFS or S3, the data can be retained for days or even months.

This is the end of the analysis on how to speed up the removal of Zookeeper in Apache Kafka 2.7.0. I hope the above content can be of some help and learn more. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.