Shulou(Shulou.com)06/03 Report--

This article mainly introduces "what is the principle of Kafka producer ack mechanism". In daily operation, I believe many people have doubts about the principle of Kafka producer ack mechanism. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the doubts about "what is the principle of Kafka producer ack mechanism?" Next, please follow the editor to study!

Partition copy

The topic of Kafka can be partitioned, and multiple copies can be configured for the partition, and the configuration can be changed through the replication.factor parameter. There are two types of partition replicas in Kafka: leader replicas (Leader Replica) and follower replicas (Follower Replica). When each partition is created, one replica is elected as the leader replica, and the remaining replicas automatically become follower replicas. In Kafka, follower copies do not provide services, that is to say, none of the follower replicas can respond to read and write requests from consumers and producers. All requests must be processed by the leader's copy. In other words, all read and write requests must be sent to the Broker where the leader's copy is located, which is handled by the Broker. The follower copy does not handle client requests, its only task is to asynchronously pull messages from the leader copy and write it to its own submission log, so as to synchronize with the leader copy.

The default replica factor for Kafka is 3, that is, there are only 1 leader copy and 2 follower copies per partition. It is shown in the following figure:

As mentioned above, the producer client only writes to the Leader broker, and the follower replicates the data asynchronously. Because Kafka is a distributed system, there must be a risk that it can not be synchronized with Leader in real time, so a method is needed to determine whether these followers have kept pace with the leaders, that is, whether the followers have synchronized the latest data. In other words, Kafka wants to tell us clearly, under what conditions can a follower copy be synchronized with Leader? This is the ISR synchronous copy mechanism that we will talk about below.

Synchronized copy (In-sync replicas)

In-sync replica (ISR) is called synchronous replica, and all replicas in ISR are synchronized with Leader, so follower that is not in the list is considered to be out of sync with Leader. So what copy exists in ISR? First of all, it is clear that a copy of Leader always exists in ISR. Whether the follower copy is in ISR depends on whether the follower copy is synchronized with the Leader copy.

Scream Tip: the understanding of "whether the follower copy is synchronized with the Leader copy" is as follows:

(1) the synchronization mentioned above does not mean complete synchronization, that is to say, once the synchronization of a follower copy lags behind that of a Leader copy, it will be kicked out of the ISR list.

(2) there is a parameter replica.lag.time.max.ms on the broker side of Kafka, which represents the maximum time interval between the lag of the follower copy and the Leader copy. The default is 10 seconds. This means that as long as the follower copy lags behind the leader copy no more than 10 seconds, the follower copy can be considered to be synchronized with the leader copy, so even if the current follower copy lags behind the Leader copy a few messages, as long as it catches up with the Leader copy within 10 seconds, it will not be kicked out.

(3) if the follower copy is kicked out of the ISR list, when the copy catches up with the progress of the Leader copy, the copy will be added to the ISR list again, so the ISR is a dynamic list, not static.

As shown in the figure above: the partition1 copy on Broker3 exceeds the specified time and is not synchronized with the Leader copy, so it is kicked out of the ISR list, and the ISR at this time is [1JE3].

Acks confirmation mechanism

The acks parameter specifies how many partition copies must receive the message before the producer considers the message to be written successfully. This parameter plays an important role in whether the message is lost. The configuration of this parameter is as follows:

Acks=0, indicating that the producer will not wait for any response from the server before successfully writing the message. In other words, once there is a problem and the server does not receive the message, the producer will not know and the message will be lost. Change the configuration because you do not need to wait for a response from the server, so you can send messages at the maximum speed supported by the network, thus achieving high throughput.

Acks=1, which means that as soon as the leader partition copy of the cluster receives the message, it will send a successful response ack to the producer. After receiving the ack, the producer can think that the message has been written successfully. Once the message cannot be written to the leader partition copy (such as network reasons, leader node crash), the producer will receive an error response. After receiving the error response, the producer will resend the data in order to avoid data loss. The throughput of this method depends on whether asynchronous or synchronous transmission is used.

Scream hint: if the producer receives an incorrect response, it is possible to lose data even if you resend the message. For example, if a node that does not receive a message becomes the new Leader, the message will be lost.

Acks = all, which means that the producer will receive a response from the server only if all the nodes participating in the replication (copies of the ISR list) receive the message. This mode is the highest level and the most secure, ensuring that more than one Broker receives the message. The latency of this mode will be very high.

Minimum synchronous copy

As mentioned above, when acks=all, all copies need to be synchronized to send a successful response to the producer. In fact, there is a question: what happens if the Leader copy is the only synchronous copy? At this time, it is equivalent to acks=1. So it's not safe.

The broker side of Kafka provides a parameter min.insync.replicas, which controls at least how many copies the message is written to before it is "really written". The default value is 1, and setting the production environment to a value greater than 1 can improve the persistence of the message. Because if the number of synchronous copies is lower than the configuration value, the producer will receive an error response to ensure that the message is not lost.

Case 1

As shown in the figure below, when min.insync.replicas=2 and acks=all, if the ISR list is only [1Power2] and 3 is kicked out of the ISR list, you only need to ensure that the two copies are synchronized, and the producer will receive a successful response.

Case 2

As shown in the figure below, when min.insync.replicas=2, if only [1], 2 and 3 of the ISR list are kicked out of the ISR list, then when acks=all, the number cannot be written successfully; when acks=0 or acks=1 can write data successfully.

Case 3

This situation can easily lead to misunderstanding. If acks=all and min.insync.replicas=2, the ISR list is [1Magne2Pol 3], then the successful response ack will not be sent to the producer until all the synchronized copies have synchronized messages. Because min.insync.replicas=2 is only a minimum limit, that is, if the synchronization copy is less than the configuration value, an exception will be thrown, while acks=all needs to ensure that all copies of the ISR list are synchronized before a successful response can be sent. As shown in the following figure:

At this point, the study on "what is the principle of Kafka producer ack mechanism" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.