Shulou(Shulou.com)06/01 Report--

This article mainly introduces "what is the principle and architecture of kafka". In daily operation, I believe many people have doubts about the principle and architecture of kafka. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful to answer the doubts about "what is the principle and architecture of kafka?" Next, please follow the editor to study!

Kafka background and Application scenario

A distributed messaging system developed by LinkedIn

High throughput

Support for message partitioning between Kafka Server and distributed consumption

Support both offline data processing and real-time data processing

Support for online horizontal scaling.

Kafka basic architecture

Explanation of kafka basic nouns

Kafka writes to disk sequentially

File Storage Mechanism of Kafka

The principle of file replication in Kafka

Guarantee of data Reliability of Kafka

Request.required.acks=-1 parameter to set the level of data reliability

Min.insync.replicas is used together.

The sending mode of Kafka. The default is synchronous producer.type=sync. Here are the asynchronous parameters.

Message Transmission guarantee of Kafka

At most once: messages may be lost, but will never be transmitted repeatedly

At least once: messages are never lost, but may be transmitted repeatedly

Exactly once: each message must be transmitted once and only once

Single Node Construction of Kafka

Download the Kafka version

Configure server.properties and consumer.properties

Kafka depends on zookeeper

Building steps

1: download kafka

Address: http://kafka.apache.org/downloads

2: upload the installation package to the server and decompress it

3: enter the config directory under the installation directory of kafka

1) Edit the server.properties file and modify the service address of zk, such as:

Zookeeper.connect=hm02:2181,hs0201:2181,hs0202:2181

2) Edit the consumer.properties file and modify the service address of zk, such as:

Zookeeper.connect=hm02:2181,hs0201:2181,hs0202:2181

4: start the zk cluster

. / zookeeper-3.4.8/bin/zkServer.sh start

5: start the server of kafka

. / bin/kafka-server-start.sh config/server.properties starts the kafka service

6: clone a session to create a topic, topic

. / bin/kafka-topics.sh-- create-- zookeeper hm02:2181,hs0201:2181,hs0202:2181-- replication-factor 1-- partitions 1-- topic mykaka

Check out those topic:

. / bin/kafka-topics.sh-list-zookeeper hm02:2181,hs0201:2181,hs0202:2181

7: create a producer, here is the producer using the command line

. / bin/kafka-console-producer.sh-broker-list localhost:9092-topic mykaka

8: create a consumer, which is also a command-line consumer

. / bin/kafka-console-consumer.sh-- bootstrap-server localhost:9092-- topic mykaka-- from-beginning

At this point, the study of "what is the principle and architecture of kafka" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.