Shulou(Shulou.com)06/02 Report--

This article mainly introduces "the basic concepts and common commands of distributed message queuing Kafka". In daily operation, I believe many people have doubts about the basic concepts and common commands of distributed message queuing Kafka. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts about "basic concepts and common commands of distributed message queuing Kafka". Next, please follow the editor to study!

Basic concept

Topic: such as database tables, or folders in the system

Partition: a topic can be divided into several partitions, and order can be guaranteed within the same partition.

Offset: an increasing integer value, the offset of each partition is unique

Broker: a standalone kafka server

MirrorMaker tool: message replication between multiple clusters

Zookeeper: save cluster metadata and consumer information, broker and topic metadata, consumer metadata partition offset

Hardware selection

Disk throughput, disk capacity, memory, network, CPU

Producer (KafkaProducer)

Serialization: custom serialization, Avro

Partition: the ProducerRecord object contains the target subject, key, and value

The key has two functions: it can be used as additional information for the message, or it can be used to determine which partition the message is rewritten to the topic, and messages with equivalent keys are written to the same partition.

Consumer (KafkaConsumer)

Consumers and consumer groups

Kafka builds data pipeline: large cache area between data segments

Kafka is a powerful message bus that can transmit event streams, but it does not have the ability to handle and transform events. The reliable delivery capability of kafka makes it a perfect data source for streaming systems.

Flume is a distributed log collection system that collects data from each server and sends it to a designated place.

Log output to flume,log4j plus log

A typical usage in the industry is:

Online data-> flume-> kafka-> hdfs-> MR offline calculation or:

Online data-> flume-> kafka-> storm

In a nutshell, flume is similar to pipes, and kafka is similar to message queues. The reason why the subject feels similar is probably because they can be used for data transmission.

Flume and Kafka should be used together, with Flume as the log collector and Kafka as the log consumer.

Flume's Source-Channel-Sink model is very suitable for log collection.

Common kafka commands:

Create topic

Bin/kafka-topics.sh-create-zookeeper localhost:2181-replication-factor 1-partitions 1-topic test

Show topic

Bin/kafka-topics.sh-list-zookeeper localhost:2181

Describe topic

Bin/kafka-topics.sh-describe-zookeeper localhost:2181-topic my-replicated-topic

Producer:

Bin/kafka-console-producer.sh-- broker-list 130.51.23.95-- topic my-replicated-topic

Consumers:

Bin/kafka-console-consumer.sh-- zookeeper 130.51.23.95topic test-- from-beginnin

At this point, the study of "basic concepts and common commands of distributed message queuing Kafka" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.