Shulou(Shulou.com)06/01 Report--

How to parse Kafka copies and ISR design, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain in detail for you, people with this need can come to learn, I hope you can gain something.

In Kafka, a partition log is actually a backup log, and kafka uses multiple identical backup logs to improve the availability of the system. These backup logs are actually called copies.

Kafka replicas can be divided into leader replicas and follower replicas, leader replicas provide read and write requests for clients, follower replicas are only used to passively synchronize data from leader replicas, and do not provide read-write services.

All Kafka nodes and all replicas are assumed to be running normally, then leader replicas will remain unchanged, but there is no absolutely stable system in the world. Once there is a problem with kafa leader replicas, then follower replicas need to compete to become leader replicas, but not all follower replicas are eligible to compete for posts. Obviously, it is ineligible to assume that the backward data of a follower is far less than that of leader replicas. Therefore, Kafka maintains a set of qualified follower copies internally, collectively referred to as ISR.

Copies in ISR will be removed and added.

Key concept points

The following figure mainly describes the important concepts in the Kafka log. The related concepts in the following figure are related to production, message consumption, ISR and replica synchronization mechanism.

First message displacement (offset): holds the offset of the first message contained in this copy

Log high watermark value (HW): the HW of the leader copy determines the range of messages that consumers can consume. Messages less than or equal to HW can be consumed by consumers.

End displacement (LEO): LEO always points to the location where the next message is written, and the message between HW and LEO of leader indicates that it has not been fully backed up. Only after all replicas in ISR have updated their own LEO, the HW of leader will move to the right to indicate that the message was written successfully.

ISR

ISR is actually a collection of replicas of synchronized follower with leader maintained internally by Kafka.

Reasons why the follower copy is out of sync with the leader copy:

The speed of synchronous data requests can not catch up: the follower copy cannot catch up with the message receiving speed of the leader replica for a period of time. For example, the network of follower replicas is blocked, which will greatly reduce the speed of follower replicas synchronizing leader replicas.

The process is stuck: the copy of follower cannot send a request to leader for a period of time, for example, follower frequently GC

Newly created replicas: users actively increase the number of copies, and the newly created replicas will catch up with the progress of leader after startup. During this period, the new follower replicas are usually out of sync with the leader replicas.

Replica.lag.max.messages

This parameter is used to detect the problem that the request speed of synchronous data can not catch up. If the number of copy messages in ISR lags behind that of leader copies exceeds the setting of this parameter, it will be kicked out of ISR.

This parameter was removed after the kafka0.9.0.9 version. Why was it removed?

There must be something wrong with him. Considering the following situation, the production rate of the producer of kafka is not stable, and there will be a peak and a low peak. At the peak, due to a large number of messages gathered, the difference between messages in ISR and messages in Leader exceeds this value, so the copy in ISR will be kicked out.

However, as the production message rate stabilizes and declines, and at this time the follower replica is also trying its best to catch up with the leader replica, when the follower replica catches up with the leader copy, it will rejoin the ISR.

Replica.lag.max.ms

This parameter is used to detect the other two situations: if the follower copy cannot request data from the leader copy during that time, it will be kicked out of the ISR.

Because the setting of the replica.lag.max.messages parameter is removed in the new version, replica.lag.max.ms is also used to detect problems in which the speed of synchronous data requests can not catch up. However, when used in the detection of secondary problems, the detection mechanism is that as long as the follower copy lags behind the leader for a period of time that does not persist beyond this parameter, it is regarded as synchronized, and if the persistence exceeds this parameter, it is regarded as out of sync.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.