Shulou(Shulou.com)06/02 Report--

What this article shares with you is about the command management of kafka. The editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

One: theme management

1: create a theme with 8 partitions and 2 replicas (the number of replicas cannot exceed the number of broker in the cluster)

Bin/kafka-topics.sh-- bootstrap-server 192.168.1.131 create-- topic mytopic-- replication-factor 2-- partitions 8

2: modify topic partition (topic partition can only be added, not deleted)

Bin/kafka-topics.sh-- bootstrap-server 192.168.1.131 alter-- topic mytopic-- partitions 10

3: delete the theme

Bin/kafka-topics.sh-- bootstrap-server 192.168.1.131 delete-- topic mytopic

4: list all topics

Bin/kafka-topics.sh-- bootstrap-server 192.168.1.131-- list

5: view topic details

Bin/kafka-topics.sh-- bootstrap-server 192.168.1.131-- describe

6: list out-of-sync topics

Bin/kafka-topics.sh-- bootstrap-server 192.168.1.131 describe-- under-replicated-partition

7: list topics without lender

Bin/kafka-topics.sh-- bootstrap-server 192.168.1.131 describe-- under-replicated-partition

Second: dynamic default configuration

1: modify the topic parameters (the message retention time is set to 1 hour, and entitiy-type topics indicates that the topic configuration is overridden)

Bin/kafka-configs.sh-zookeeper 192.168.1.131 alter-entity-type topics-entity-name mytopic-add-config retention.ms=3600000

2: modify the default configuration of brokers

Bin/kafka-configs.sh-- bootstrap-server 192.168.1.131 alter-- entity-type brokers-- entity-name 0-- add-config num.io.threads=10

3: modify the default configuration of the client

Bin/kafka-configs.sh-zookeeper 192.168.1.131 alter-entity-type clients-entity-name admin-add-config producer_bytes-rate=10

4: remove dynamic configuration

Bin/kafka-configs.sh-- zookeeper 192.168.1.131 alter-entity-type topics-entity-name mytopic-delete-config retention.ms

5: view dynamic configuration

Bin/kafka-configs.sh-- zookeeper 192.168.1.131 describe-entity-type topics-entity-name mytopic

Third: partition management (Kafka provides two script management partitions, one for re-election of leaders and one for assigning partitions to broker. With the combination of these two tools, you can achieve load balancing of cluster traffic. Using multiple partition replicas can improve reliability, but only one of them can become the partition leader, and only the broker where the leader is located can carry out production and consumption activities. Kafka selects the first synchronized copy of the replica list as the leader, but does not automatically restore the original leader status after shutting down and restarting broker. Broker has a configuration that can be used to enable automatic leader rebalancing, but this feature is not recommended in production environments. )

1: start the partition election (kafka default election)

Bin/kafka-preferred-replica-election.sh-- zookeeper 192.168.1.131purl 2181

2: start the partition election and specify the election leader (written in the configuration file)

Bin/kafka-preferred-replica-election.sh-zookeeper 192.168.126.128purl 2181-path-to-json-file partitons.json

3: modify the partition copy

Sometimes you need to modify the partition copy. Below, you need to modify the scene of the partition copy.

1. The uneven distribution of topic partitions in the whole cluster results in the unbalanced load of the cluster.

2.broker offline causes out of sync of partitions

3. The new broker needs to get the load from the cluster.

You can use the Kafka-reassign-partitions.sh tool to modify the partition.

Step 1: generate a set of migration steps based on the broker list and the topic list

Step 2, perform the migration steps

Step 3: you can use the generated migration steps to verify the progress and completion of partition reassignment.

Example: generate migration steps for themes in the topics.json file and migrate them to broker0 and broker1.

Bin/kafka-reassign-partitions.sh-- zookeeper 192.168.1.131 generate-topics-to-move-json-file topics.json-broker-list 0pen 1

This tool will output two json on the console, describing the current partition and the recommended partition allocation scheme. You can save the first json for rollback.

The second json should be saved to another file as input to the kafka-reassign-partitons.sh tool to perform the second step.

Example: use reassign.json to implement the recommended partition allocation scheme

Bin/kafka-reassign-partitions.sh-- zookeeper 192.168.1.131 execute-reassignment-json-file reassign.json

This command reassigns a copy of the specified partition to the new broker.

You can use the kafka-reassign-partitions.sh tool to verify the status of the redistribution during or after the redistribution is in progress.

Example: verify the partition reallocation specified in the reassign.json file.

Bin/kafka-reassign-partitions.sh-- zookeeper 192.168.1.131 verify-reassignment-json-file reassign.json

The above is what the command management of kafka is. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.