Shulou(Shulou.com)06/02 Report--

Today, I would like to talk to you about the mechanism by which Kafka is kept highly reliable and highly available. Many people may not know much about it. In order to make you understand better, the editor has summarized the following contents for you. I hope you can get something from this article.

This Acks parameter is a very core and key parameter in the use of Kafka, which determines a lot of things.

The following analysis of the Acks parameters of Kafka, as well as the principle behind it.

How to ensure that data is not lost in case of downtime?

If you want to understand the meaning of this Acks parameter, you must first understand the highly available architecture principles of Kafka.

For example, the figure below shows that for each Topic, we can set it to contain several Partition, and each Partition is responsible for storing part of the Topic data.

Then in the Broker cluster of Kafka, some Partition is stored on each machine, that is, part of the data of Topic is stored, so that the data of Topic is distributed and stored on a Broker cluster.

But there is a problem, if a Kafka Broker goes down, won't the data stored above be lost?

Yes, this is a big problem. The problem of data loss in distributed systems must be solved first. Once any machine goes down, it will lead to data loss.

Highly available mechanism of multi-copy redundancy

So if you analyze the principles of any distributed system, such as Zookeeper, Kafka, Redis Cluster, Elasticsearch, HDFS, and so on.

In fact, they all have their own internal mechanism of multi-copy redundancy, which is a common function of almost any excellent distributed system.

In a Kafka cluster, each Partition has multiple replicas, one of which is called Leader and the other is called Follower, as shown in the following figure:

As shown in the figure above, suppose a Topic is split into three Partition, which is Partition0,Partiton1,Partition2, and each Partition has two copies.

For example, one copy of Partition0 is Leader, and the other is Follower,Leader and Follower. The two copies are distributed on different machines.

Such a multi-copy redundancy mechanism can ensure that any machine crashes, will not lead to complete loss of data, because at least there are copies on other machines.

How to synchronize data between multiple replicas?

Then let's take a look at how data is synchronized between multiple copies. In fact, any Partition, only Leader provides external reading and writing services.

That is, if a client writes data to a Partition, it usually writes a Leader copy of that Partition.

Then after the Leader copy receives the data, the Follower copy will constantly send it requests to try to pull the * * data, pull it locally, and write it to disk.

As shown in the following figure:

What exactly does ISR mean?

Now that you know how Partiton synchronizes data with multiple copies, let's take a look at what ISR is.

The full name of ISR is "In-Sync Replicas", that is, a copy that stays in sync, and it means what kind of Follower is always in sync with Leader.

You can think about this: if a certain Broker of Follower is stuck because of problems such as JVM FullGC and cannot pull synchronized data from Leader in time, will the data of Follower lag far behind that of Leader?

So at this point, it means that Follower is no longer in a synchronized relationship with Leader.

But as long as Follower synchronizes data from Leader all the time, they can be guaranteed to be in a synchronous relationship.

So every Partition has an ISR, and there must be Leader itself in this ISR, because the Leader is sure that the data is *, and then even the Follower that is synchronized with the Leader will be in the ISR.

The meaning of Acks parameter

With so much laying the groundwork, * * can finally get to the topic and talk about the meaning of the Acks parameter.

If you do not understand the previous copy mechanism, synchronization mechanism, ISR mechanism, then you will not fully understand the meaning of the Acks parameter, which actually determines a lot of important things.

First of all, the Acks parameter is set in Kafka Producer, that is, the producer client.

In other words, you can set this Acks parameter when you write data to Kafka. Then this parameter actually has three common values that can be set: 0, 1, and all.

* the option is to set the Acks parameter to 0, which means that my Kafka Producer is on the client side. As long as the message is sent, regardless of whether the data is on disk on the Partition Leader or not, I will ignore it and directly think that the message has been sent successfully.

If you use this setting, it is important to note that the message you send may still be halfway there.

As a result, the Broker where the Partition Leader is located hangs directly, and then your client still thinks that the message was sent successfully, which will cause the message to be lost.

The second option is to set Acks = 1, which means that as long as the Partition Leader receives the message and writes it to the local disk, it is considered successful, regardless of whether its other Follower synchronizes the past message or not.

This kind of setting is actually the default setting of Kafka, please pay attention to it and highlight it! This is the default setting.

In other words, by default, if you ignore the parameter Acks, you will be considered successful as long as the Partition Leader is successful.

However, there is a problem. If the Partition Leader has just received the message and the Follower has not had time to synchronize it, as a result, the Broker where the Leader is located is down, and the message will be lost because the client already thinks that the message has been sent successfully.

* one case is to set Acks=all, which means that after Partition Leader receives the message, it must also require those Follower in the ISR list that are synchronized with Leader to synchronize the message before it can be considered to be written successfully.

If Partition Leader has just received the message, but as a result Follower has not received the message, and Leader is down, then the client will perceive that the message has not been sent successfully, and he will try to send the message again.

At this point, the Follower of Partition2 may become Leader, and only the Follower of * * in the ISR list will be transformed into Leader, so as long as the new Leader receives the message successfully.

Thinking

Can Acks=all represent that the data will not be lost? Of course not. If you have only one copy of your Partition, that is, a Leader, and you don't have any Follower, do you think acks=all is useful?

Of course, it doesn't work, because there is only one Leader in ISR, and it goes down after receiving messages, which will also lead to data loss.

So, this Acks=all must be used with at least 2 copies in the ISR list, at least one Leader and one Follower.

Only in this way can you ensure that if you write a piece of data in the past, it must be a success if more than 2 copies are received. At this time, any copy is down, which will not lead to data loss.

After reading the above, do you have any further understanding of the mechanism by which Kafka maintains high reliability and high availability? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.