Installation and Management of kafka Cluster (2)

I. Migration of broker

1. View zookeeper and kafka startup

[root@slave1] # pssh-h hostlist-I 'jps' [1] 22:08:11 [SUCCESS] 20.0.5.123492 QuorumPeerMain8140 Jps6414 Kafka [2] 22:08:11 [SUCCESS] 20.0.5.133490 QuorumPeerMain4972 Kafka22972 Jps [3] 22:08:11 [SUCCESS] 20.0.5.117369 QuorumPeerMain23754 Jps11534 Kafka [4] 22:08:11 [SUCCESS] 20.0.1421263 Jps [5] 22:08:11 [SUCCESS] 20.0.5.1520818 Jps

two。 Create a topic

[root@slave1] # / opt/kafka/bin/kafka-topics.sh-- create-- zookeeper 20.0.5.11 zookeeper 2181-- replication-factor 3-- partitions 3-- topic topic1Created topic "topic1". [root @ slave1 ~] # / opt/kafka/bin/kafka-topics.sh-- describe-- zookeeper 20.0.5.11 zookeeper 2181-- topic topic1Topic:topic1 PartitionCount:3 ReplicationFactor:3 Configs: Topic:topic1 Partition: 0 Leader: 1 Replicas: 1p3 2 Isr: 1,3,2 Topic: topic1 Partition: 1 Leader: 2 Replicas: 2,1,3 Isr: 2,1,3 Topic: topic1 Partition: 2 Leader: 3 Replicas: 3,2,1 Isr: 3,2,1

3. Start the kafka process on the node to be migrated

[root@slave1] # pssh-h hostlist'/ opt/kafka/bin/kafka-server-start.sh / opt/kafka/config/server.properties > / root/kafka.log 2 > 1 [1] 22:17:21 [SUCCESS] 20.0.5.12 [2] 22:17:21 [SUCCESS] 20.0.5.11 [3] 22:17:21 [SUCCESS] 20.0.5.13 [4] 22:17:21 [SUCCESS] 20.0.5.14 [5 ] 22:17:21 [SUCCESS] 20.0.5.15 [root@slave1 ~] # pssh-h hostlist-I 'jps' [1] 22:17:29 [SUCCESS] 20.0.5.123492 QuorumPeerMain8461 Jps6414 Kafka [2] 22:17:29 [SUCCESS] 20.0.1421298 Kafka21609 Jps [3] 22:17:29 [SUCCESS] 20.0.5.117369 QuorumPeerMain24844 Jps11534 Kafka [4] 22:17:29 [SUCCESS] 20.0.5.133490 QuorumPeerMain4972 Kafka23292 Jps [5] 22: 17:29 [SUCCESS] 20.0.5.1520854 Kafka21165 Jps

4. Create json files for topic that need to be migrated

[root@slave1 ~] # cat topic_move.json {"topics": [{"topic": "topic1"}], "version": 1}

5. Generate migration rules

[root@slave1 ~] # / opt/kafka/bin/kafka-reassign-partitions.sh-- zookeeper 20.0.5.11 zookeeper 2181-- topics-to-move-json-file topic_move.json-- broker-list "3jue 4jue 5"-- generateCurrent partition replica assignment {"version": 1, "partitions": [{"topic": "topic1", "partition": 1, "replicas": [2jue 1jue 3], "log_dirs": ["any", "any", "any"]} {"topic": "topic1", "partition": 0, "replicas": [1 version 2], "log_dirs": ["any", "any", "any"]}, {"topic": "topic1", "partition": 2, "replicas": [3 replicas 2], "log_dirs": ["any", "any", "any"]} Proposed partition reassignment configuration {"version": 1, "partitions": [{"topic": "topic1", "partition": 1, "replicas": [5pr 3 4], "log_dirs": ["any", "any", "any"]}, {"topic": "topic1", "partition": 0, "replicas": [4 any 5 any 3], "log_dirs": ["any", "any", "any"]}, {"topic": "topic1", "partition": 2, "replicas": [3pr 4pr 5], "log_dirs": ["any", "any", "any"]}

6. Write the generated data to a new json file

[root@slave1 ~] # cat re_node.json {"version": 1, "partitions": [{"topic": "topic1", "partition": 1, "replicas": [5pc3pas 4], "log_dirs": ["any", "any", "any"]}, {"topic": "topic1", "partition": 0, "replicas": [4pr 5pr 3], "log_dirs": ["any", "any", "any"]}, {"topic": "topic1", "partition": 2 "replicas": [3Jing 4jue 5], "log_dirs": ["any", "any", "any"]}]}

7. Start the migration

[root@slave1 ~] # / opt/kafka/bin/kafka-reassign-partitions.sh-- zookeeper 20.0.5.11 executeCurrent partition replica assignment 2181-- reassignment-json-file re_node.json-- executeCurrent partition replica assignment {"version": 1, "partitions": [{"topic": "topic1", "partition": 1, "replicas": [2Jing 1jue 3], "log_dirs": ["any", "any", "any"]}, {"topic": "topic1", "partition": 0, "replicas": [1Jue " 2], "log_dirs": ["any", "any", "any"]}, {"topic": "topic1", "partition": 2, "replicas": [3 any 2 1], "log_dirs": ["any", "any" "any"]} Save this to use as the-- reassignment-json-file option during rollbackSuccessfully started reassignment of partitions. [root@slave1 ~] # / opt/kafka/bin/kafka-reassign-partitions.sh-- zookeeper 20.0.5.11 reassignment-json-file option during rollbackSuccessfully started reassignment of partitions 2181-- reassignment-json-file re_node.json-- verifyStatus of partition reassignment:Reassignment of partition topic1-1 completed successfullyReassignment of partition topic1-0 completed successfullyReassignment of partition topic1-2 completed successfully

8. View the new partition

[root@slave1] # / opt/kafka/bin/kafka-topics.sh-- describe-- zookeeper 20.0.5.11 zookeeper 2181-- topic topic1Topic:topic1 PartitionCount:3 ReplicationFactor:3 Configs: Topic:topic1 Partition: 0 Leader: 4 Replicas: 5 Isr: 5 Topic:topic1 Partition 3 Topic:topic1 Partition: 1 Leader: 5 Replicas: 5 Replicas 3 Isr: 5 Replicas 3 4 Topic: topic1 Partition: 2 Leader: 3 Replicas: 3,4,5 Isr: 5,3,4

# modify producer and consumer to point to a new connection after migration, and then stop the old service