Shulou(Shulou.com)06/01 Report--

Today, I will talk to you about how to analyze reblance in Kafka. Many people may not know much about it. In order to make you understand better, the editor has summarized the following contents for you. I hope you can get something according to this article.

The common consumption patterns of Kafka are organized in groups. Usually, Kafa distributes the partitions of Topic evenly to different instances under the same group. There are three common strategies:

Range: arranges all partitions of a single Topic in order, then divides these partitions into fixed size segments and assigns them to each consumer. Default policy

Round: assign all Topic partition polling for subscriptions to each conumser

Sticky: avoid data skew and maximize the allocation scheme before the maintenance of the two reblance

Currently, reblance can be triggered in the following ways:

Group members change: new consumer joins and leaves the group, consumer crashes unexpectedly

The number of Topic for group subscriptions changes: for example, subscriptions based on regular expressions, when a new Topic matching regular expressions is created

When the number of partitions of the Topic subscribed by the group changes

Reblance generation

Consumer group can execute reblance multiple times. In order to protect consumer group, especially from invalid offset commits, reblance generation is usually used to identify a reblance, which is incremented by 1 each time reblance is experienced. The default value starts at 0. If a consumer with a generation value of 1 has a delayed commit, but the reblance has generated a new group member and the generation value has changed to 2, then the submission of the conumse will be ILLEGAL_EXCEPTION.

Reblance protocol

Kafka uses the following four sets of requests to complete the reblance.

JoinGroup:consumer request to join a group

SyncGroup:group leader synchronously updates the allocation scheme to all members of the group

HeartBeat:consumer reports regularly to coordinator that his heartbeat shows that he is still alive.

LeaveGroup:consumer actively asked coordinator to leave the group himself.

In addition to the above four sets of requests, there is a special request:

DescribeGroup: view all the information about the group, including membership information, protocol information, allocation scheme, subscription information, etc. This request does not participate in reblance and is mainly used by administrators.

During the reblance process, coordinator needs to receive JoinGroup and SyncGroup requests from consumer. When reblance is successful, consumer sends HeartBeat requests to coordinator periodically. Consumer will also determine whether the current group has opened a new round of reblance based on whether REBLANCEINPROCESS is included in the HeartBeat response. When consumer actively leaves the group, a LeaveGroup request needs to be sent to coordinator.

Reblance process

Before consumer reblance, you need to select the broker where the coordinator is located (and establish a Socket connection). Algorithm:

Math.abs (groupId.hashCode) offsets.topic.num.partitions.

Reblance is mainly divided into two steps:

Join the group: all consumer in the group sends JoinGroup requests to coordinator. When all JoinGroup requests are collected, coorinator needs to select a group leader and send all member information and their subscription information to leader.

Synchronous update allocation scheme: group leader is responsible for allocating consumption plans. There are three specific strategies at the beginning of the article. After the allocation is complete, leader encapsulates the allocation scheme into an SyncGroup request and sends it to coordinator. In this step, all consumer sends SyncGroup requests, except that only leader contains the allocation scheme. After receiving the request, coordinator extracts the consumption information of each consumer and sends it to the corresponding consumer as a response of SyncGroup.

After reading the above, do you have any further understanding of how to analyze reblance in Kafka? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.