Shulou(Shulou.com)06/01 Report--

Today, I will talk to you about how to increase the number of topic backups in kafka. Many people may not know much about it. In order to make you understand better, the editor has summarized the following for you. I hope you can get something according to this article.

I. difficulties and difficulties

When you set up a topic, you can set the number of backups by specifying the parameter, replication-factor. However, once the topic has been established, the number of replica cannot be modified through kafka-topic.sh or the command.

II. Solutions

In fact, we can consider an "alternative" approach: you can use the kafka-reassign-partitions.sh command to redistribute all partitions, and when doing partition redistribution, you can achieve this by increasing the number of replica backups for each partition.

This article describes how to use the kafka-reassign-partitions.sh command to increase the number of backups for topic.

Note: the following command uses the topic name, the ip and port of zookeeper, and requires the reader to replace it with the parameters of the actual cluster.

(suppose there are four broker,id in the kafka cluster: 1001, 1002, 1003, 1004)

2.1 、 Get all the partitions of the current topic distributed in the broker [root@tbds bin] #. / kafka-topics.sh-- zookeeper 172.16.32.13 zookeeper 2181-- topic ranger_audits-- describeTopic:ranger_audits PartitionCount:10 ReplicationFactor:1 Configs: Topic:ranger_audits Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001 Topic:ranger_audits Partition: 1 Leader: 1002 Replicas: 1002 Isr: 1002 Topic: ranger_audits Partition: 2 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: ranger_audits Partition: 3 Leader: 1002 Replicas: 1002 Isr: 1002 Topic: ranger_audits Partition: 4 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: ranger_audits Partition: 5 Leader: 1002 Replicas: 1002 Isr: 1002 Topic: ranger_audits Partition: 6 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: ranger _ audits Partition: 7 Leader: 1002 Replicas: 1002 Isr: 1002 Topic: ranger_audits Partition: 8 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: ranger_audits Partition: 9 Leader: 1002 Replicas: 1002 Isr: 1002

As you can see, the ranger_audits topic has 10 partitions, each with only one feplica backup, distributed on 1001 and 1002 broker.

Next we need to increase each partition data of ranger_audits to 2 replica backups and distribute them to 4 broker.

2.2.Create configuration files that increase the number of replica backups

(note: try to keep the original primary backup of each partition of topic unchanged. Therefore, the first broker of each partition of the configuration file remains the same. )

[root@tbds bin] # vim.. / config/increase-replication-factor.json {"version": 1, "partitions": [{"topic": "ranger_audits", "partition": 0, "replicas": [1001J 1003]}, {"topic": "ranger_audits", "partition": 1, "replicas": [1002m 1004]}, {"topic": "ranger_audits", "partition": 2, "replicas": [1001m 1003]}, {"topic": "ranger_audits", "partition": 3 "replicas": [1002jue 1004]}, {"topic": "ranger_audits", "partition": 4, "replicas": [1001jue 1003]}, {"topic": "ranger_audits", "partition": 5, "replicas": [1002jue 1004]}, {"topic": "ranger_audits", "partition": 6, "replicas": [1001prime1003]}, {"topic": "ranger_audits", "partition": 7, "replicas": [1002jue 1004]}, {"topic": "ranger_audits", "partition": 8 "replicas": [1001jue 1003]}, {"topic": "ranger_audits", "partition": 9, "replicas": [1002meme1004]}]}

According to the above configuration file, we added one replica to each partition of topic, kept the original primary backup broker of each partition unchanged, and put the new replica backup data of each partition on the 1003 and 1004 broker.

2.3. start adding partition [root@tbds bin] #. / kafka-reassign-partitions.sh-zookeeper 172.16.32.13 reassignment-json-file 2181-- reassignment-json-file.. / config/increase-replication-factor.json-- executeCurrent partition replica assignment {"version": 1, "partitions": [{"topic": "ranger_audits", "partition": 3, "replicas": [1002]}, {"topic": "ranger_audits", "partition": 9, "replicas": [1002]} {"topic": "ranger_audits", "partition": 8, "replicas": [1001]}, {"topic": "ranger_audits", "partition": 1, "replicas": [1002]}, {"topic": "ranger_audits", "partition": 4, "replicas": [1001]}, {"topic": "ranger_audits", "partition": 2, "replicas": [1001]}, {"topic": "ranger_audits", "partition": 5, "replicas": [1002]} {"topic": "ranger_audits", "partition": 0, "replicas": [1001]}, {"topic": "ranger_audits", "partition": 6, "replicas": [1001]}, {"topic": "ranger_audits", "partition": 7, "replicas": [1002]}} Save this to use as the-- reassignment-json-file option during rollbackSuccessfully started reassignment of partitions {"version": 1, "partitions": [{"topic": "ranger_audits", "partition": 0 "replicas": [1001jue 1003]}, {"topic": "ranger_audits", "partition": 8, "replicas": [1001jue 1003]}, {"topic": "ranger_audits", "partition": 5, "replicas": [1002Mague 1004]}, {"topic": "ranger_audits", "partition": 2, "replicas": [1001Pert 1003]}, {"topic": "ranger_audits", "partition": 9, "replicas": [1002jue 1004]}, {"topic": "ranger_audits", "partition": 1 "replicas": [1002jue 1004]}, {"topic": "ranger_audits", "partition": 3, "replicas": [1002jue 1004]}, {"topic": "ranger_audits", "partition": 4, "replicas": [1001Mague 1003]}, {"topic": "ranger_audits", "partition": 7, "replicas": [1002prime1004]}, {"topic": "ranger_audits", "partition": 6 "replicas": [1001meme1003]} 2.4.View the execution progress [root@tbds bin] #. / kafka-reassign-partitions.sh-zookeeper 172.16.32.13 kafka-reassign-partitions.sh 2181-reassignment-json-file.. / config/increase-replication-factor.json-- verifyStatus of partition reassignment:Reassignment of partition [ranger_audits,0] completed successfullyReassignment of partition [ranger_audits,8] completed successfullyReassignment of partition [ranger_audits,5] completed successfullyReassignment of partition [ranger_audits,2] completed successfullyReassignment of partition [ranger_audits 9] completed successfullyReassignment of partition [ranger_audits,1] completed successfullyReassignment of partition [ranger_audits,3] completed successfullyReassignment of partition [ranger_audits,4] completed successfullyReassignment of partition [ranger_audits,7] completed successfullyReassignment of partition [ranger_audits,6] completed successfully

It shows that the operation of adding partition was successful.

2.5 、 Check the situation of topic again [root@tbds bin] #. / kafka-topics.sh-- zookeeper 172.16.32.13 kafka-topics.sh 2181-- topic ranger_audits-- describeTopic:ranger_audits PartitionCount:10 ReplicationFactor:2 Configs: Topic:ranger_audits Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001 Topic:ranger_audits Partition: 1 Leader: 1002 Replicas: 1002 Topic:ranger 1004 Isr: 1002 _ audits Partition: 2 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003 Topic: ranger_audits Partition: 3 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004 Topic: ranger_audits Partition: 4 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003 Topic: ranger_audits Partition: 5 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004 Topic: ranger_audits Partition: 6 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003 Topic: ranger_audits Partition: 7 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004 Topic: ranger_audits Partition: 8 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003 Topic: ranger_audits Partition: 9 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004

As can be seen from the above, the number of backups has increased successfully.

Third, further thinking

Using the method described above, in addition to increasing the number of topic backups, you can also handle the following scenarios:

* * 1. Migrate all partition data of topic. * * how to understand it? If the cluster has N broker, then M broker will be expanded. As the newly expanded broker disks are empty, the original broker disks are full. Then we can use the above method to move some topic stored in the original N broker to the newly expanded M broker, so as to achieve the overall data balance of the kafka cluster.

The specific usage is as follows: by writing the configuration file of chapter 2.2, configure all the partitions of topic to the new M broker, and then execute excute, you can complete the overall migration of all partition data of topic to the newly expanded M broker nodes.

* 2. The situation where broker is broken. * causes a decrease in the number of replica in some topic partitions, and kafka-reassign-partitions.sh can be used to increase replica.

* 3, kafka some broker disks are full and some disks are small. * kafka-reassign-partitions.sh can be used to migrate some topic partition data to broker with less disk footprint to achieve data balance.

* 4. Kafka cluster expansion. * you need to migrate the topic data of the original broker to the new broker, and make rational use of the newly expanded broker to achieve load balancing.

After reading the above, do you have any further understanding of how to increase the number of topic backups in kafka? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.