Shulou(Shulou.com)06/01 Report--

This article mainly introduces "Kafka installation tutorial". In daily operation, I believe many people have doubts about the installation tutorial of Kafka. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts of "installation tutorial of Kafka". Next, please follow the editor to study!

Basic terminology of 1.Kafka

Producer: the message producer, which is the client that sends messages to kafka broker

Consumer: message consumer, the client that fetches messages from kafka broker

Topic: topic, which can be understood as a queue

Consumer Group (CG): this is the means used by kafka to broadcast (to all consumer) and unicast (to any consumer) a topic message. A topic can have more than one CG. Messages from topic are copied (not really copied, but conceptually) to all CG, but each CG sends messages to only one consumer in that CG. If you need to implement broadcasting, as long as each consumer has a separate CG. To achieve unicast, all consumer are in the same CG. With CG, you can also group consumer freely without having to send messages to different topic multiple times.

Broker: a kafka server is a broker. A cluster consists of multiple broker. A broker can hold multiple topic.

Partition: to achieve scalability, a very large topic can be distributed across multiple broker (that is, servers), a topic can be divided into multiple partition, and each partition is an ordered queue. Each message in partition is assigned an ordered id (offset). Kafka only guarantees that messages are sent to consumer in the order in one partition, not the order of a topic as a whole (among multiple partition).

The storage files of Offset:kafka are named after offset.kafka, and the advantage of using offset as a name is that it is easy to find. For example, if you want to find the location at 2049, just find the 2048.kafka file. Of course, the first offset is 00000000000.kafka.

two。 Install Kafka

# # download

[grid@hadoop4 ~] $wget http://mirrors.cnnic.cn/apache/kafka/0.8.2.1/kafka_2.11-0.8.2.1.tgz

[grid@hadoop4 ~] $tar-zxf kafka_2.11-0.8.2.1.tgz

# # modify $KAFKA_HOME/config/server.properties

[grid@hadoop4] $cd kafka_2.11-0.8.2.1

[grid@hadoop4 kafka_2.11-0.8.2.1] $vim config/server.properties

# # mainly modify the following configurations: broker.id, port, host.name, log.dirs, zookeeper.connect

# # for the installation of zookeeper cluster, please refer to http://my.oschina.net/zc741520/blog/399233

3.Kafka basic command

Start kafka bin/kafka-server-start.sh. / config/server.properties

Create topic bin/kafka-topics.sh-- topic kafkaToptic-- create-- zookeeper 127.0.0.1 create-- replication-factor 1-- partition 1

View consumer bin/kafka-console-consumer.sh-- zookeeper 127.0.0.1-- topic kafkaToptic-- from-beginning

View topic bin/kafka-topics.sh-- list-- zookeeper 127.0.0.1purl 2181

Production message bin/kafka-console-producer.sh-- broker-list 127.0.0.1 topic kafkaToptic

4. Start Kafka (need to start Zookeeper first)

[grid@hadoop4 kafka_2.11-0.8.2.1] $bin/kafka-server-start.sh. / config/server.properties &

5. Create a topic named apache_log

[grid@hadoop4 kafka_2.11-0.8.2.1] $bin/kafka-topics.sh-topic apache_log-create-zookeeper hadoop4:2181-replication-factor 1-partition 1 &

[grid@hadoop4 kafka_2.11-0.8.2.1] $bin/kafka-topics.sh-- list-- zookeeper hadoop4:2181

Apache_log

At this point, the study on the "installation tutorial of Kafka" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.