Shulou(Shulou.com)05/31 Report--

How to analyze kafka data preservation time and kafka performance testing, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain in detail for you, people with this need can come to learn, I hope you can get something.

Check the topic of kafka. Bin/kafka-topics.sh-- describe-- zookeeper hadoop234:2182-- topic test5

You can view the leader of replicationfactor. If a node in broker list is broken, leader will be re-selected.

Partition: partition

Leader: the node responsible for reading and writing the specified partition

Replicas: copy the list of nodes for the partition log

Isr: "in-sync" replicas, the currently active replica list (which is a subset), and may become Leader.

In the process of sending a message by producer, a node of the broker list is cut off, which will not affect the sending of the message.

Kafka data preservation time problem.

There are two main types of Kafka deletion checks, which are performed as soon as they meet the requirements.

(1) minutes or hours can be set according to time granularity. Time is reached for processing.

(2) set the maximum file size according to the file size limit, and process it as soon as it reaches the upper limit.

The file size check period can be set.

Performance Test of kafka

Kafka has its own performance test code, and the test results come from the test code that comes with kafka, which is located in bin/kafka-producer-perf-test.sh.

The main parameters are as follows

Total number of messages sent by messages producer

Message-size size of each message (in b)

Number of messages sent in batches by batch-size

Topic sent by topics producer

Threads producers use several threads to send simultaneously

For example, bin/kafka-producer-perf-test.sh-- messages 100000-- message-size 1000-- batch-size 10000-- topics test4-- threads 4-- broker-list hadoop234:9092,hadoop237:9092,hadoop238:9092

The returned information is

Start.time, end.time, compression, message.size, batch.size, total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec

2015-10-15 18-56-26-27-52, 2015-10-15-- 56-18-18-15-- 56-18-30, 880, 0, 1000, 10000, 95.37, 28.5702, 100000, 29958.0587

Performance testing can be carried out by adjusting the parameters.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.