Shulou(Shulou.com)06/01 Report--

This article mainly introduces how to view Kafka Topic consumption, has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let Xiaobian take you to understand.

1. Purpose of document writing

When our application consumes Topic messages from Kafka, we will pay attention to the total amount of messages and consumption in Topic. Then Fayson will mainly introduce how to use the command line in the CDH cluster to view Topic consumption.

The test environment for this Fayson is

Version 5.15 for 1.CM and CDH

2.Redhat7.4

3.Kafka 0.10.2+kafka2.2.0

two。 Command line view

The script for kafka-run-class.sh is provided in the script of Kafka, which is used to view the Offset of consuming Topic in the Consumer group.

1. Execute the following command on the command line to view the consumption of the specified Topic and ConsumerGroup

[root@cdh01 bin] # kafka-run-class kafka.tools.ConsumerOffsetChecker-zookeeper cdh01.fayson.com:2181-group test-topic kafka_kudu_topic_offset_3

For the first time, check the Offset consumption of each Partition in the testgroup group of kafka_kudu_topic_offset_3 and the total number of messages per Partition.

Topic of 2.kafka_kudu_topic_offset_3 sends 600 messages and executes the command again to check the consumption.

When the applications that are not consumed are running, you can see that the data of Offset has not changed, while the data of logSize has increased, and the number of each partition has increased by 200, which is exactly the number of messages I sent.

3. Start the consumer program to consume the Topic message, and execute the command again to check the consumption situation

When the consumer application is running, the Offset data of each Partition changes. At this time, the Offset data is consistent with the logSize data, and there are no new messages received in the application log, indicating that the Topic messages have been consumed by the testgroup group.

Thank you for reading this article carefully. I hope the article "how to check the Topic consumption of Kafka" shared by the editor will be helpful to everyone. At the same time, I also hope that you will support and follow the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.