Shulou(Shulou.com)05/31 Report--

This article introduces how the offset mechanism Offset in the core of Kafka is, the content is very detailed, interested friends can use it for reference, I hope it can be helpful to you.

Kafka is a real-time data processing framework open source by LinkIn, which has been updated to version 2.3. Different from the general message middleware, Kafka achieves extremely high throughput through data persistence and disk reading and writing, and can carry out real-time stream processing on its own without relying on Storm,SparkStreaming 's stream processing platform.

Kakfa's Offset mechanism is one of its core mechanisms, because API for the implementation of some functions, we sometimes do not manually set Offset, so how many Offset does Kafka have?

1. Producer Offset

First of all, let's take a look at the producer's offset. We know that Kafka is a message sent by the producer to a certain topic, and then the consumer consumes the topic message. Of course, there may be multiple producers, multiple consumers, and there may also be the concept of consumer group, which will be discussed later.

When a producer sends a message to a topic, it depends on how many partitions there are, because the kafka is distributed through a partitioning mechanism.

Map producer offset

Through this diagram, we can clearly see the offset principle of producers, no matter how many producers, or we specify which partition they will write, but as long as they write, each partition must have an offset, this offset is the producer's offset, but also the latest and largest offset of this partition.

Sometimes when we develop the producer code, we don't specify the offset of a particular partition, maybe we use a single partition, or we write multiple partitions evenly by default. Kafka helps us do this.

II. Consumer Offset

Let's take a look at the consumer-side offset, which is a little more complicated.

Figure Consumer offset

This is the case of the offset of a certain partition. We already know that the offset written by the producer is the latest and largest value, that is, 12. When Consumer A consumes, he consumes from 0 to 9, and his offset is recorded in 9. Consumer B records 11. The next time they come to spend, they can choose to continue their last consumption, of course, they can choose to spend from scratch, or they can skip to the recent record and start spending "now".

In this way, consumers can use it flexibly even if there are multiple partitions.

Figure Consumer Group

In fact, the concept of consumer group does not affect the understanding of offset. In the above case, if Consumer A feather Consumer B is in the same group, it cannot consume the same partition message at the same time. Consumers in different groups can consume the same partition message at the same time.

There is also a saying of offset, that is, when the consumer consumption is not submitted, there is another offset locally, which is not necessarily consistent with the offset recorded in the cluster.

Therefore, kafka has multiple offset for each topic partition and producer, and the consumer is different.

Offset refers to the offset of a partition.

Topic partition offset, these three are the only ones that confirm a message.

The producer's offset is actually the latest offset.

The consumer's offset is maintained by himself, and he can choose to partition the beginning, the latest, and also remember where he is spending.

Consumer groups are designed so that consumers in different groups can consume a partition message at the same time.

On the Kafka core of the offset mechanism Offset is shared here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.