Shulou(Shulou.com)06/02 Report--

This article mainly explains "what is the Leader election in Kafka". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Let's let Xiaobian take you to learn "What is the Leader election in Kafka"!

A message is considered submitted only if all followers in the ISR have copied past the Leader. This prevents some data from being written into the Leader, and before it can be copied by any Follower, it goes down, causing data loss. For Producer, it can choose whether to wait for the message commit, which can be set by request.required.acks. This mechanism ensures that as long as there is one or more followers in the ISR, a committed message will not be lost.

What is ISR?

A very important question is how to elect a new Leader in the Followers when the Leader is down, because the Followers may fall behind a lot or crash directly, so make sure to choose the latest Follower as the new Leader. A basic principle is that if the Leader is no longer there, the new Leader must have all the messages of the original Leader commit. This requires a trade-off, if the Leader waits for more Followers to acknowledge a message before it is committed, then more Followers can become new Leaders after it dies, but this also causes throughput to drop.

A very common way of electing leaders is "minority obeys majority," which Kafka does not use. In this mode, if we have 2f+1 replicas, we must ensure that f+1 replicas have copied the message before committing, and the number of failed replicas cannot exceed f in order to ensure that the new Leader can be elected correctly. This approach has the advantage that the latency of the system depends on the fastest machines, i.e., if the number of copies is 3, then the latency depends on the fastest Follower rather than the slowest. The "minority obeys the majority" method also has some disadvantages. In order to ensure the normal operation of the Leader election, it can tolerate a relatively small number of failed Followers. If one Follower is to be tolerated, at least three copies are required. If two Followers are to be tolerated, at least five copies are required. In other words, in order to ensure a high fault tolerance rate in a production environment, there must be a large number of replicas, and a large number of replicas will lead to a sharp decline in performance under large data volumes. This algorithm is used more often in systems like Zookeeper that share a clustered configuration than in systems that require large amounts of data. HDFS's HA function is also based on a "minority obeys the majority" approach, but its data storage does not adopt this approach.

In fact, there are many algorithms for Leader election, such as Zookeeper's Zab, Raft and Viewstamped Replication. Kafka's Leader selection algorithm is more like Microsoft's Pacific A algorithm.

Kafka dynamically maintains an ISR for each Partition in Zookeeper. All replicas in this ISR keep up with the Leader, and only members in the ISR can be selected as the Leader (uncleader.leader.election.enable=false). In this pattern, for f+1 copies, a Kafka Topic can tolerate f copies failing without losing committed messages, which is advantageous in most usage scenarios. In fact, in order to tolerate f copies of failure, the "majority" approach requires the same number of copies as the ISR must wait before committing, but the total number of copies required by the ISR is almost half that of the "majority" approach.

As mentioned above, when there is at least one Follower in the ISR, Kafka can ensure that the committed data is not lost, but if all replicas of a Partition are dead, there is no guarantee that the data will not be lost. In this case, there are two possible options:

Wait for any replica in ISR to come alive and select it as Leader

Select the first replica that comes alive (not necessarily in ISR) as Leader

This requires a simple choice between availability and consistency. If you have to wait for the replica in ISR to "come alive," the unavailability time may be relatively long. And if all replicas in ISR fail to "come alive" or data is lost, the Partition will never be available. Select the first replica that comes alive as the Leader, and this replica is not a replica in ISR, even if it does not guarantee that all the committed messages have been included, it will become the Leader and serve as the data source for the Consumer. By default, Kafka uses the second policy, unclean.leader.election.enable=true, or you can set this parameter to false to enable the first policy.

uncleader.leader.election.enable This parameter has a critical impact on leader selection, system availability, and data reliability. Below we analyze some typical scenarios.

If the number of replicas in a Partition is 3, replica-0, replica-1, replica-2 are stored in Broker0, Broker1, and Broker2, respectively. AR=(0,1,2)，ISR=(0,1)。Set request.required.acks=-1, min.insync.replicas=2, unclean.leader.election.enable=false. The copy in Broker0 is also referred to as Broker 0 initially as Leader and Broker1 as Follower.

When replica-0 in ISR crashes, Broker1 elects the new Leader[ISR=(1)] because write cannot be serviced due to min.insync.replicas=2, but read can continue to be serviced normally. Recovery programmes for such cases:

Try to recover (restart)replica-0, if it can get up, the system is normal;2. If replica-0 cannot be restored, you need to set min.insync.replicas to 1 to restore the write function.

When replica-0 in ISR crashes, then replica-1 crashes, at this time [ISR=(1),leader=-1], can not provide services to the outside world, in this case recovery scheme:

Try to restore replica-0 and replica-1, if both can get up, then the system returns to normal;

If replica-0 is up, but replica-1 cannot be up, then the Leader cannot be elected, because when unleader.leader.election.enable=false is set, the Leader can only be elected from the ISR. When all copies in the ISR are invalid, the Leader can only be elected after the last invalid copy in the ISR can be recovered, that is, replica-0 is invalid first, replica-1 is invalid, and replica-1 is recovered before the Leader can be elected. A conservative solution suggests setting unclean.leader.election.enable=true, but this will cause data loss and restore the read service. Also set min.insync.replicas to 1 to restore the write function;

Replica-1 is restored, replica-0 cannot be restored. This situation has been encountered above. The read service is available. You need to set min.insync.replicas to 1 to restore the write function.

Neither replica-0 nor replica-1 can be recovered, which can be referred to Case 2.

When replica-0 and replica-1 in ISR are down at the same time, at this time [ISR=(0,1)], no external service can be provided. In this case, the recovery scheme: try to recover replica-0 and replica-1. When any one of them recovers normally, read service can be provided externally. Write functionality will not resume until 2 copies are restored, or min.insync.replicas will be set to 1.

At this point, I believe everyone has a deeper understanding of "what is the Leader election in Kafka". Let's do it in practice. Here is the website, more related content can enter the relevant channels for inquiry, pay attention to us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.