Shulou(Shulou.com)06/01 Report--

This article mainly explains "what are the core points of kafka tuning". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn what are the core points of kafka tuning.

1 Kafka pressure test

Kafka officially comes with stress test scripts (

Kafka-consumer-perf-test.sh, kafka-producer-perf-test.sh). During the Kafka stress test, you can see where there is a bottleneck (CPU, memory, network IO). Generally speaking, the network IO reaches the bottleneck.

Number of machines with 2 Kafka

Number of Kafka machines = 2 * (peak production speed * copies / 100) + 1

3 Log save time of Kafka

7 days

4 hard disk size of Kafka

Amount of data per day * 7 days

5 Kafka monitoring

The monitor developed by the company itself

Open source monitors: KafkaManager, KafkaMonitor

6 number of Kakfa partitions

The number of partitions is not as large as possible. In general, the number of partitions should not exceed the number of cluster machines. The more partitions, the more memory (ISR, etc.), the more partitions in a node set, and the greater the impact on the system when it goes down.

The number of partitions is generally set to 3-10

7 set the number of copies

In general, we set it to 2 or 3, and many enterprises set it to 2.

8 how many Topic

Typically, there are as many Topic as there are log types. There are also combinations of log types.

9 Kafka can't lose data

Ack=0, which is equivalent to sending asynchronously, increases offset as soon as the message is sent, and continues production.

When Ack=1,leader receives leader replica's acceptance of a message, ack adds offset, and then continues production.

Ack=-1,leader receives all the replica's acceptance of a message ack before adding offset, and then continues production.

10 Kafka ISR copy synchronization queue

ISR (In-Sync Replicas), copy synchronization queue. Leader and Follower are included in ISR. If the Leader process dies, a service is selected in the ISR queue as the new Leader. There are two parameters, replica.lag.max.messages (delay number) and replica.lag.time.max.ms (delay time), which determine whether a service can join the ISR copy queue. In version 0.10, the replica.lag.max.messages parameter is removed to prevent the service from joining the queue frequently.

If any dimension exceeds the threshold, the Follower will be removed from the ISR and stored in the OSR (Outof-Sync Replicas) list, and the newly added Follower will be stored in the OSR first.

11 Kafka partition allocation policy

There are two default partition allocation strategies within Kafka: Range and RoundRobin.

Range is the default policy. Range is for each Topic (that is, one Topic and one Topic). First, the partitions in the same Topic are sorted by sequence number, and the consumers are sorted alphabetically. Then divide the number of Partitions partitions by the total number of consumer threads to determine how many partitions each consumer thread consumes. If not, the first few consumer threads will consume an extra partition.

For example: we have 10 partitions, two consumers (C1 ~ C2), 3 consumer threads, 10 / 3 = 3 and inexhaustible.

C1-0 will consume 0, 1, 2, 3 zones

C2-0 will consume 4, 5, 6 zones

C2-1 will consume 7, 8, 9 zones.

RoundRobin: premise: all consumers in the same Consumer Group must have equal num.streams (consumer consumption threads); each consumer must subscribe to the same topic.

At this point, I believe you have a deeper understanding of "what are the core points of kafka tuning?" you might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.