Shulou(Shulou.com)06/01 Report--

This article mainly explains "what are the basic knowledge points of Kafka". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "what are the basic knowledge points of Kafka".

1. Installation and deployment 1.1, cluster planning Hadoop102Haoop103Hadoop104ZookeeperZookeeperZookeeperkafkakafkakafka1.2, jar package download

Http://kafka.apache.org/downloads

1.3 、 # globally unique number of broker Cannot repeat broker.id=2-need to modify # number of threads processing network requests num.network.threads=3 # off-the-shelf number of disk IO num.io.threads=8 # buffer size of sending sockets socket.send.buffer.bytes=102400 # receiving sockets Buffer size socket.receive.buffer.bytes=102400 # request socket buffer size socket.request.max.bytes=104857600 # kafka run log storage path log.dirs=/opt/module/kafka/logs-- the number of partitions # topic on the current broker needs to be modified num.partitions=1 # to recover and clean up the data Number of threads of data maximum retention time of num.recovery.threads.per.data.dir=1 # segment files Timeout will be removed log.retention.hours=168 # configure connection Zookeeper cluster address zookeeper.connect=hadoop102:2181,hadoop103:2181 Hadoop104:2181/kafka-- need to modify 5) configure environment variable sudo vim / etc/profile.d/my_env # KAFKA_HOME export KAFKA_HOME=/opt/module/kafka export PATH=$PATH:$KAFKA_HOME/bin source / etc/profile.d/my_env6) distribute installation package xsync kafka/ Note: remember to configure the environment variables of other machines after distribution Modify broker.id=3 in the configuration file / opt/module/kafka/config/server.properties on hadoop103 and hadoop104, respectively. Broker.id=4 Note: broker.id must not be repeated! 7) start the cluster in hadoop102, hadoop103, Start Kafka [atguigu @ hadoop102 kafka] $kafka-server-start.sh-daemon $KAFKA_HOME/config/ server. Properties [atguigu @ hadoop103 kafka] $kafka-server-start.sh-daemon $KAFKA_HOME/config/server.properties [atguigu@hadoop104 kafka] $kafka-server-start.sh-daemon $KAFKA_HOME/config/server.properties8) on the hadoop104 node to close the cluster [atguigu@hadoop102 kafka] $bin/kafka-server-stop.sh [atguigu@hadoop103 kafka] $bin/kafka-server-stop.sh [ Atguigu@hadoop104 kafka] $bin/kafka-server-stop.sh9) kafka group stop script! # / bin/bash#1, Determine the number of parameters if [$#-ne 1] Then echo "args num is errorproof!" Exitfi#2, execute logic case $1 in "start") for host in hadoop02,hadoop103,hadoop104 do echo "= start $host kafka=" ssh $host "/ opt/module/kafka/bin/kafka-server-start.sh-daemon / opt/module/kafka/config/server.properties" done according to the entered parameters "stop") for host in hadoop102,hadoop103,hadoop104 do echo "= stop $host kafka=" ssh $host "/ opt/module/kafka/bin/kafka-server-stop.sh" done (stop,start,status) " Esac2, Kafka Command Line Operation 2.1 View all topickafka-topics.sh in the current server-- list-- bootstrap-server hadoop102:9092,hadoop103:9092,hadoop104:90922.2 create topickafka-topics.sh-- create-- bootstrap-server hadoop102:9092 Hadoop103:9092-- topic second-- partitions 2-- replication-factor 2 option description:-- topic definition topic name-- number of replication-factor definition copies-- partitions definition number of partitions 2.3.Delete topickafka-topics.sh-- delete-- bootstrap-server hadoop102:9092 Hadoop103:9092-- topic second2.4 send message kafka-console-producer.sh-- broker-list hadoop102:9092-- topic first2.5 consumption message kafka-console-consumer.sh-- bootstrap-server hadoop102:9092-- from-beginning-- topic first--from-beginning: will read all the previous data in the topic. 2.6 check the details of a Topic kafka-topics.sh-- describe-- bootstrap-server hadoop102:9092,hadoop103:9092-- topic first2.7 modify the number of partitions kafka-topics.sh-- alter-- bootstrap-server hadoop102:9092,hadoop103:9092-- topic first-- partition 6 Note: partitions can only be changed to larger, not smaller, not the number of copies. Thank you for reading. These are the contents of "what are the basic knowledge points of Kafka?" after the study of this article. I believe that you have a deeper understanding of what are the basic knowledge points of Kafka, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.