How to parse kafka

How to analyze kafka, I believe that many inexperienced people are at a loss about it. Therefore, this paper summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

What is kafka?

First of all, we need to understand the requirements under which kafka is generated. With the continuous rise of big data in recent years, social networking, search, business activities have produced a huge amount of data, and the problems we are facing also follow. How should we collect this huge amount of information? Can we collect these data in time? What are we going to do with it after collection?

These problems give rise to a new business model: producers and consumers. The producers of data are called producers (users, enterprises), and the processors of data are called consumers (NoSQL,Hadoop,). So how to solve the problem of information transmission between producers and consumers? Kafka is the framework to solve the problem of information transmission between producers and consumers, and realizes the seamless docking of information.

Second, the characteristics of kafka:

1. High-output distributed message system

2. A single kafka service can process hundreds of MB data sent by thousands of clients per second.

3. A single cluster can be used as a big data processing center to deal with various types of business.

4. Messages are persisted to disk (can handle TB data level data but still maintain high data processing efficiency), and there is a backup fault tolerance mechanism.

5. The information produced can be consumed by consumers immediately.

III. Components of kafka

1. Topics and Logs: the directory where messages are stored is the topic. Kafka cluster divides each topic into multiple partition (zones). Each partition is an ordered and immutable message sequence, which exists in the form of append log files at the storage level. Any messages published to this partition are appended directly to the end of the log file. The position of each message in the file is called offset (offset), and offset is a long number, which uniquely marks a message.

2. Producer: the side of the production message to the topic. Producer sends the message in the specified topic, and the producer decides which partition to send the message to.

3. Consumer: the party who subscribes to topic consumption messages. There are usually two modes of message delivery, queuing (queue, where each Consumer takes a message from the message queue) and publish-subscribe (publish-subscribe, where messages are broadcast to each Consumer)

4. A Broker:Kafka cluster contains one or more servers.

After reading the above, do you know how to parse kafka? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!