Shulou(Shulou.com)06/01 Report--

This article mainly explains "what is the water level of Kafka". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "what is the water level of Kafka".

What is the water level?

Water level is described in kafka.

The offset of visible data in a partition.

Maybe you need to know these things:

Hw (water level) you can understand it as a global (minimum offset of all replicas) offset

For a partition.

LEO represents the maximum offset of all messages in this copy

It's for one copy, that is, each copy has LEO, and it's different.

The smallest LEO of all copies is the water level.

Water level

Why the water level?

To ensure the consistency of the data

The Operation of Water level in Kafka

When Leader receives a message and writes it successfully

Its LEO is and immediately + 1.

The copy will be synchronized with Leader regularly.

For each message synchronized, its own LEO corresponds to + 1

And the water level is the smallest LEO of all copies.

So it will increase slowly.

The relationship with ACK

In fact, there is no inevitable relationship.

ACK is used to ensure that data is not lost.

Hw is used to ensure the consistency of client consumption.

But when ACK=-1

Because it will wait until the data is fully written to all copies.

To return to success.

In other words, the LEO of all copies is + 1

Its water level must also be + 1.

At this time, data loss can be avoided.

Otherwise, there is a risk of data loss.

Why ACK! =-1 data may be lost

For example, one partition, four copies, one, two, two, three, four:

1:LEO = 10 (Leader)

2:LEO = 8

3:LEO = 7

4:LEO = 6

At this point hw = 6

When No.1 Leader is dead.

The 3rd was elected as the new Leader.

Then all surviving copies will first erase all data except hw.

Then synchronize the data after the water level from Leader

For example, copy 2:

Clear your own data first. 7, 8, and Leo becomes 6.

Then go to Leader to synchronize the data after 6.

Even if the old Leader 1 recovers, the 6-10 data is still there.

You also need to clear all the data after the hw during the downtime.

Resynchronize data from Leader

Thank you for your reading, the above is the content of "what is the water level of Kafka". After the study of this article, I believe you have a deeper understanding of what the water level of Kafka is, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.