Shulou(Shulou.com)06/03 Report--

Tips: To see the HD uncoded set of pictures, please use your mobile phone to open and click on the picture to enlarge.

1. problem description

Deleting topics with the kafka-topics --delete command does not actually delete them, but marks them as "marked for deletion," causing recreations of topics with the same name to report the error "already exists."

2. problem recurrence

1. Log in to the server where the Kafka cluster is located and create a test topic.

[root@cdh2 ~]# kafka-topics --create --zookeeper ip-172-31-1-175.ap-southeast-1.compute.internal:2181 --replication-factor 1 --partitions 1 --topic test

2. View newly created topics

[root@cdh2 ~]# kafka-topics --list --zookeeper ip-172-31-1-175.ap-southeast-1.compute.internal:2181

3. Delete the new topic by issuing the following command

[root@cdh2 ~]# kafka-topics --delete --zookeeper ip-172-31-1-175.ap-southeast-1.compute.internal:2181 --topic test

"Topic test is marked for deletion"

4. Try to recreate a Topic for test

[root@cdh2 ~]# kafka-topics --create --zookeeper ip-172-31-1-175.ap-southeast-1.compute.internal:2181 --replication-factor 1 --partitions 1 --topic test

Prompt that the Topic already exists, indicating that the deletion in step 3 does not actually delete the Topic.

3. cause of the problem

By default, Kafka disables topic deletion, so when performing Topic deletion, it only marks the deleted Topic as "marked for deletion". It can be enabled by modifying the configuration parameters of the Kafka service.

4. solution

4.1 Method 1

In the case of kafka service configuration delete.topic.enable=false, if you need to permanently delete topic, you need to do the following:

Delete the corresponding topic by kafka command Delete the corresponding topic in zookeeper Delete the log data of topic on the broker node where topic is located

The operation is as follows:

1. To view the description of topic, command as follows

kafka-topics --describe --zookeeper ip-172-31-1-175.ap-southeast-1.compute.internal:2181 --topic test

The data storage node corresponding to topic in the marked part of the figure above

2. Remove by kafka command, command as follows:

kafka-topics --delete --zookeeper ip-172-31-1-175.ap-southeast-1.compute.internal:2181 --topic test

3. Delete topic through zookeeper client, command as follows

zookeeper-client -server ip-172-31-1-175.ap-southeast-1.compute.internal:2181

ls /brokers/topics

rmr /brokers/topics/test

Note: The part marked in red is the name of the topic

4. Log in to the log data directory of the topic of the corresponding node listed in step 1. Here, our log.dirs directory of Kafka is configured as/var/local/kakfa, and execute the command.

[ec2-user@ip-172-31-9-186 data]$ sudo rm -rf test-0/

5. Verification is effective

Create a Topic named "test", which can be created normally

Note: After deleting the log directory with topic test (/var/local/kafka/test-0), the newly created log directory with topic test does not exist. After restarting the Kafka service, it works normally and the directory can be displayed normally.

4.2 Method 2

When Kafka service has configured delete.topic.enable=true, permanently deleting topic requires the following actions:

Delete topic using kafka command

The operation is as follows:

Data view before deletion:

kafka-topics --describe --zookeeper ip-172-31-1-175.ap-southeast-1.compute.internal:2181 --topic test

Information about each data node:

Zookeeper information:

2. Delete the topic to be deleted by kafka command, as follows

kafka-topics --delete --zookeeper ip-172-31-1-175.ap-southeast-1.compute.internal:2181 --topic test

3. View Zookeeper client topic information

[zk: ip-172-31-1-175.ap-southeast-1.compute.internal:2181(CONNECTED) 9] ls /brokers/topics

4. View log data for topic on brober node

5. After successful deletion, verify that a Topic named test can be created

Note:

If Kafka service is configured with delete.topic.enable=true, delete it directly from command line. If Topic cannot be deleted, delete it through zookeeper-client.

If Kafka service is not configured with delete.topic.enable=true, delete topic directly by using delete command. When deleting topic, it will only be marked as "marked for deletion", and then delete it through zookeeper-client. If the data.log data directory of topic will not be deleted, you need to delete the corresponding topic directory under the data.log directory on the corresponding broker server. The data.log directory can be obtained, which can be viewed through CM interface:

Suggestion: Kafka service is enabled delete.topic.enable=true, and the way to enable it is as follows:

Kafka service needs to be restarted after startup.

Drunken whip famous horse, youth more pompous! Lingnan Huanxi Sand, vomiting wine shop! Friends refused to put, data play flowers!

Tips: To see the HD uncoded set of pictures, please use your mobile phone to open and click on the picture to enlarge.

Welcome to pay attention to Hadoop practice, the first time, share more Hadoop dry goods, like please pay attention to share.

Original article, welcome to reprint, please indicate: Reprint from Weixin Official Accounts Hadoop Practical Operation

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.