Shulou(Shulou.com)06/03 Report--

This article introduces the relevant knowledge of "how to configure KRaft". In the operation process of actual cases, many people will encounter such difficulties. Next, let Xiaobian lead you to learn how to deal with these situations! I hope you can read carefully and learn something!

1. How do I get started with KRaft?

Kafka replaces ZooKeeper with an embedded KRaft, which is a big step forward because distributed systems like ES, which synchronize cluster meta information, are self-looping.

But how do I start it with KRaft? A lot of students are dizzy dish directly, the data of this respect also is less, but use rises very simple.

We noticed that under the config directory, there is an additional directory called kraft, which contains a new set of configuration files that can directly eliminate the dependency on ZK.

With the following three commands, you can open a standalone broker without ZK's participation from the beginning to the end.

# ./ bin/kafka-storage.sh random-uuid # ./ bin/kafka-storage.sh format -t TBYU7WMiREexuZqrjKG60g -c ./ config/kraft/server.properties # ./ bin/kafka-server-start.sh ./ config/kraft/server.properties

After a series of crackling runs, No ZK's Kafka has been activated.

It's that simple.

2. How is it configured?

Kafka adds an internal theme called @metadata to store this metadata.

Next we will look at some key configuration information. You can use vimdiff config/server.properties config/kraft/server.properties to see the main differences.

First, kraft adds a configuration called process.roles. It looks like this in our profile.

process.roles=broker,controller

It actually has three values.

broker: This machine will be treated as just a broker

controller: Start as one of the controllers of Raft quorum

broker,controller: contains both functions

Those familiar with ES can see that these divisions are like the master and node of ES, so the concept of distribution is actually interlinked to some extent.

Next is the change of listening address, because our server has two functions, so we need to open two ports.

listeners=PLAINTEXT://:9092,CONTROLLER://:9093

There is also something called node. id. Unlike the original broker.id, this nodeid is used for voting.

node.id=1

Because of the nature of the raft protocol, our voting configuration uses node.id above. It's kind of weird, isn't it? But it looks better than ZK. So these configurations are subject to change in later versions.

controller.quorum.voters=1@localhost:9093

This is the main difference between profiles. Let's look at its collection.

process.roles=broker,controller listeners=PLAINTEXT://:9092,CONTROLLER://:9093 node.id=1 controller.quorum.voters=1@localhost:90933. Why kill ZK?

As a message queue, Kafka relies on a heavyweight coordination system ZooKeeper, which has to be said to be a joke. Also as a message queue, RabbitMQ realized self-management early on.

Zookeeper is very heavy and requires an odd number of nodes in the cluster configuration. It is also inconvenient to expand and shrink capacity. Zk's configuration method was completely different from Kafka's. To optimize Kafka according to Kafka, he actually had to consider another system. This was really f * cking dog.

If Kafka wanted to go lightweight and out-of-the-box, it had to kill Zk.

In addition, since Zk and Kafka are not in a storage system after all, when the number of topics and partitions increases, the data synchronization problem becomes significant. Zk is reliable, but it is slow, completely inferior to Kafka's log storage system, which is a link that Kafka has to bypass.

Students who have used Kafka-admin should have a vivid memory of slow monitoring data synchronization. It needs to go around zk, get some metadata information, and then pull data from Kafka's JMX interface. This kind of wandering almost killed the large cluster.

4. What will change?

Deployment is simpler.

First, deployment becomes easier. For some systems that aren't so high-availability driven, even a single process can get the cute kafka running. We also don't need to apply for zookeeper friendly SSD disks, and we don't need to worry about whether zk's capacity is enough.

Monitoring is easier.

Second, because of the concentration of information, it is easy to get surveillance information from Kafka without having to go to zk again. Integration with systems such as grafana/kibana/promethus is imminent.

Faster.

The most important thing, of course, was speed. Raft is easier to understand and more efficient than ZK's ZAB protocol, partition's main election will be faster, and controller scheduling will be a notch faster.

In the future, there will never be such a connection.

zookeeper.connect=zookeeper:2181

Instead, only bootstrap connections remain. Kafka's nodes are becoming more and more like peer nodes.

bootstrap.servers=broker:9092

Kafka also provides a tool called kafka-metadata-shell.sh that allows you to see the distribution of topics and partions, information that was previously available via zk but is now available from this command line.

$ ./ bin/kafka-metadata-shell.sh --snapshot /tmp/kraft-combined-logs/\@metadata-0/00000000000000000000.log >> ls / brokers local metadataQuorum topicIds topics >> ls /topics foo >> cat /topics/foo/0/data { "partitionId" : 0, "topicId" : "5zoAlv-xEh9xRANKXt1Lbg", "replicas" : [ 1 ], "isr" : [ 1 ], "removingReplicas" : null, "addingReplicas" : null, "leader" : 1, "leaderEpoch" : 0, "partitionEpoch" : 0 } >> exit"How to configure KRaft" content is introduced here, thank you for reading. If you want to know more about industry-related knowledge, you can pay attention to the website. Xiaobian will output more high-quality practical articles for everyone!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.