Shulou(Shulou.com)05/31 Report--

In this article, the editor introduces in detail how to use the message partition allocation algorithm in kafka. The content is detailed, the steps are clear, and the details are handled properly. I hope this article "how to use the message partition allocation algorithm in kafka" can help you solve your doubts.

Background

Kafka has a partitioning mechanism, and a topic topic sets partitions when it is created. If there is only one partition, then all consumers subscribe to this partition message; if there are multiple partitions, how are consumers distributed?

Allocation algorithm

RangeAssignor definition

Kafka defaults to the allocation algorithm of RangeAssignor.

The principle of the RangeAssignor strategy is to obtain a span according to the total number of consumers and the total number of partitions, and then distribute the partitions evenly according to the span to ensure that the partitions are distributed to all consumers as evenly as possible. For each Topic,RangeAssignor policy, all consumers in the consumption group who subscribe to this Topic are sorted according to the lexicographic order of their names, and then each consumer is divided into a fixed range of partitions. If it is not evenly distributed, consumers with the top dictionary order will be assigned more partitions.

The obvious problem with this allocation is that with the increase in the number of Topic subscribed by consumers, the imbalance will become more and more serious. For example, in the scenario of 4 partitions and 3 consumers in the figure above, C0 will allocate more partitions. If you subscribe to Topic with 4 partitions at this time, C0 will allocate more partitions to C1 and C2, so that C0 will allocate two more partitions to C1 and C2, and the situation will become more and more serious with the increase of Topic.

Source code analysis public class RangeAssignor extends AbstractPartitionAssignor {.... Override public Map > assign (Map partitionsPerTopic, Map subscriptions) {/ / 1. Get how many consumer subscriptions to each topic Map consumersPerTopic = consumersPerTopic (subscriptions); / / 2. Store the final allocation file Map assignment = new HashMap (); for (String memberId: subscriptions.keySet ()) assignment.put (memberId, new ArrayList ()); for (Map.Entry > topicEntry: consumersPerTopic.entrySet ()) {String topic = topicEntry.getKey (); List consumersForTopic = topicEntry.getValue (); / / 3. The number of partition per topic Integer numPartitionsForTopic = partitionsPerTopic.get (topic); if (numPartitionsForTopic = = null) continue; Collections.sort (consumersForTopic); / / 4. Indicates how many partition int numPartitionsPerConsumer = numPartitionsForTopic / consumersForTopic.size () will be allocated to each consumer; / / 5. How many partition are left unallocated after average distribution int consumersWithExtraPartition = numPartitionsForTopic% consumersForTopic.size (); List partitions = AbstractPartitionAssignor.partitions (topic, numPartitionsForTopic); / / 6. This is the key point, and the allocation principle is to allocate the partition that cannot be evenly distributed to the first consumersWithExtraPartition consumer for (int I = 0, n = consumersForTopic.size (); I

ConsumersWithExtraPartition? 0: 1); assignment.get (consumersForTopic.get (I)) .addAll (partitions.subList (start, start + length));}} return assignment;}} scene

It can be distributed evenly.

It can not be distributed evenly, and the ranking is higher than that of the previous ones.

The number of consumers is greater than the number of districts, ranking at the top and getting points first, but not divisions at the bottom of the rankings.

RoundRobinAssignor definition

The allocation strategy of RoundRobinAssignor is to allocate all the partitions of Topic subscribed to within the consumption group and all consumers in a balanced order (RangeAssignor is allocated for the partition of a single Topic). If the consumer subscribes to the same Topic list within the consumer group (each consumer subscribes to the same Topic), the distribution result is as balanced as possible (the difference in the number of partitions allocated between consumers will not exceed 1).

Source code analysis package org.apache.kafka.clients.consumer; public class RoundRobinAssignor extends AbstractPartitionAssignor {@ Override public Map > assign (Map partitionsPerTopic, Map subscriptions) {assignment = new HashMap (); for (String memberId: subscriptions.keySet ()) assignment.put (memberId, new ArrayList ()); / / 1. Circular linked list, which stores all the consumer, "after each iteration" will return to the origin CircularIterator assigner = new CircularIterator (Utils.sorted (subscriptions.keySet (); / / 2. Get the total number of partition of all subscribed topic for (TopicPartition partition: allPartitionsSorted (partitionsPerTopic, subscriptions)) {final String topic = partition.topic (); while (! subscriptions.get (assigner.peek ()). Topics (). Contains (topic) assigner.next (); assignment.get (assigner.next ()) .add (partition);} return assignment;}. } scene

It can not be distributed evenly, and the ranking is higher than that of the previous ones.

StickyAssignor definition

Although RoundRobinAssignor has made some optimizations on RangeAssignor to allocate partitions more evenly, in some cases, it will still produce serious distribution deviations. In the word sense, Sticky is "sticky", which can be understood as "sticky"-each allocation change makes the least change compared to the previous allocation (the previous result is sticky). It is marked with two points:

The distribution of zones is as balanced as possible.

The result of each redistribution is as close as possible to that of the previous one.

Scene

After reading this, the article "how to use the message partition allocation algorithm in kafka" has been introduced. If you want to master the knowledge of this article, you still need to practice and use it to understand it. If you want to know more about related articles, welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.