Shulou(Shulou.com)06/01 Report--

Kafka kernel distributed mechanism implementation is how, I believe that many inexperienced people are helpless, for this reason this article summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

All data in a Topic is stored distributively on all brokers in the kafka cluster, and data is stored in the form of partitions; backup data/backup partitions (partitions stored on other brokers in the same kafka cluster) are allowed in each partition.

Each data partition in the Kafka cluster has a partition on the broker node called leader, and backup partitions stored on other brokers are called followers; only the leader node is responsible for data read and write operations of the partition, and the followers node acts as a hot standby node for the leader node, backing up data from the leader node; when the leader node hangs, one of the followers nodes will become the leader node and provide services again.

Kafka Cluster Partition leader and followers switch dependent on Zookeeper

The first feature of Kafka's distributed assurance is Kafka's Replication.

Kafka's Replication refers to the replication of Partition, only one partition of all partitions of a Partition is the leader node, and the other partitions are the follower nodes.

Replication has some impact on Kafka throughput, but greatly enhances availability

Follower nodes periodically fetch incremental data from leader nodes. An active follower node must satisfy the following two conditions:

1. All nodes must maintain connections to zookeeper (via zk's heartbeat)

2. follower must be able to copy the writing on the leader in a timely manner, and cannot "lag too much"; "lag too much" is determined by parameters {replica.lag.time.max.ms} and {replica.lag.max.messages}

The second feature of Kafka distributed assurance is Kafka Leader Election.

Kafka provides an in-sync replicas(ISR) to ensure Kafka's Leader election. ISR is a collection of nodes that save partitions. If a node goes down or the data is "too late," the leader will remove the node from ISR. Only the follower node in ISR can become the leader node.

The switching of the Leader node is based on Zookeeper's Watcher mechanism. When the leader node goes down, the follower nodes in other ISRs will compete to create a file directory in zk (only one follower node will be successfully created). The successfully created follower node becomes the leader node.

After reading the above, do you know how to implement the distributed mechanism in Kafka kernel? If you still want to learn more skills or want to know more related content, welcome to pay attention to the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.