Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "what is the role of Controller in Kafka". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

In addition to state management, Kafka has the following main functional responsibilities:

Update cluster metadata information

Create Topic

Delete Topic

Partition redistribution

Preferred leader copy election

Topic partition extension

Broker joins the cluster

Broker crashes

Controlled shutdown

Controller leader election

Responsibility function updates cluster metadata information

Client can obtain the complete metadata information of the cluster from any broker, which requires controller to notify each broker when the cluster metadata information changes. When the partition information changes, controller encapsulates the changed information in a UpdateMetadataRequest request and sends it to each Broker in the cluster.

Create Topic

When Controller starts, it creates a Zookeeper listener that listens for changes in child nodes under / brokers/topics.

When Topic is created, it first creates a child node (znode) under the / brokers/topics node, and writes the partition and copy of the Topic to the znode. When the listener discovers that there is a new znode in / brokers/topics, it immediately triggers controller to create a Topic (identify Leader and ISR for each partition of the new Topic), and then update the cluster metadata information.

When the creation is complete, controller creates a new listener / brokers/topics/ {topic_name} to listen for changes to the new Topic content. In this way, when the Topic partition changes, the controller can be notified in time.

Delete Topic

Kafka cluster deletes Topic by adding a child node (znode) under / admin/deletetopics under Zookeeper. When controller starts, it creates a listener in / admin/deletetopics to listen to whether the children of that node have changed. When a new child node is created, controller immediately turns on the delete Topic delete logic. There are two main steps to implement:

Stop all copies of the Topic

Delete copy log data

Controller removal / admin/delete_topics/ {topic node to be deleted}, topic deletion completed

Partition redistribution

The main purpose of partition redistribution is to reallocate the broker machine where the replica resides to all partitions under the Topic in order to achieve a more uniform distribution. This operation requires the administrator to manually customize the allocation scheme and write it in the specified format under / admin/reassign_partitions of Zookeeper.

The process of partition redistribution is a process of expanding and then shrinking. Controller first extends partition replicas, sets Leader as replicas in the new allocation scheme after they are all synchronized with Leader replicas, and finally performs contraction to reduce partition replicas to a collection of replicas in the allocation scheme.

Preferred leader election

In order to avoid uneven distribution of partition replicas, the concept of preferred replicas is introduced. Assuming that the list of partitioned replicas is [1 ~ 2 ~ 3], then broker1 is the preferred leader of the partition. But with the passage of time, the partition leader changes, and the final preferred leader is not necessarily the partition leader.

Broker segment parameter auto.leader.reblance.enable setting true,controller automatically adjusts preferred leader at regular intervals

Triggered by kafka-preferred-replica-election script

Both of the above methods write data to the / admin/preferredreplicaelection node of Zookeeper. Controller also registers the node's directory listener. Once a change notification is received, controller adjusts the leader of the corresponding partition back to the first one in the replica list and broadcasts it.

Topic partition extension

Currently, additional partitions are usually added through the-alert option of kafka-topics.sh, which writes a new partition directory to / brokers/topics/ {Topic} under Zookeeper.

Since a listener listening for directory changes is registered after the Topic is created, controller is notified when the directory changes, performs the partition creation task, and then updates the cluster metadata information.

Broker joins the cluster

After each broker is created successfully, a child node is created under the / brokers/ids node of the Zookeeper and the information of the broker is written. Controller creates a listener at / brokers/ids to listen for changes in the children of that node, and once notified, the cluster metadata information is updated.

Broker crashes

The node registered by broker when joining the cluster is a temporary node. With the crash of broker, the temporary node disappears, and then controller is notified to turn on and off the broker logic and update the cluster metadata information.

Controlled shutdown

Controlled shutdown refers to shutting down kafka broker in the form of kafka-server-stop.sh or kill-15.

A controlled shutdown is a ControlledShutdownRequest sent to controller by the soon-to-be-closed broker. When the request is sent, broker is blocked, and controller will send ControlledShutdownResoponse to broker after leader re-election and ISR contraction adjustment, indicating that broker can be closed.

Controller leader election

When all the broker starts together, the temporary / controller node will be created on the Zookeeper first, and the information of the controller broker will be written. Zookeeper ensures that only one broker can be created successfully, and the successful broker node will update the value of / controller_epoch. If you do not create a successful broker snooping / controller, join the controller and go down, and other broker will be notified and run for controller.

Communication between controller and broker

When controller starts, it establishes TCP connections to all broker in the cluster, including the broker where controller is located, and establishes a RequestSendThread for each TCP connection, which means that controller establishes a TCP connection to each broker and starts an TCP O thread.

Currently, controller has the following three main requests:

UpdateMetaRequest: update the cluster metadata request, which contains the cluster metadata information. When broker receives the request, it updates the local cache information

LeaderAndIsrRequest: for creating partitions and replicas

StopReplicaRequest: stops the data request operation of the specified copy and is also responsible for deleting the replica data function.

This is the end of the content of "what is the role of Controller in Kafka". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.