Shulou(Shulou.com)06/02 Report--

This article mainly introduces "diagrams of basic concepts in Kafka". In daily operation, I believe that many people have doubts about the diagrams of basic concepts in Kafka. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful to answer the doubts of "diagrams of basic concepts in Kafka". Next, please follow the editor to study!

This time you will learn the basic concepts of Kafka. First of all, let's review that in the usage scenario of the messaging system, there are three roles: the producer, the messaging system and the consumer, in which the producer is responsible for generating and sending messages to the messaging system, and the messaging system will provide messages for consumers for processing, as shown in the following figure.

Kafka is a messaging system based on the publish / subscribe model, as shown in the following figure. The producer pushes the message to a Topic in the Kafka. The purpose of introducing Topic is to classify messages so that consumers can subscribe to the corresponding Topic to get messages according to their needs.

Although Kafka was able to start working in this way, it was followed by a single point of problem. In order to solve the single point problem, Kafka introduces the concept of Broker. A Broker is an Kafka instance, while multiple Broker can be run on a machine, here we think that there is only one Kafka instance on a machine. Multiple Broker will form a Kafka cluster, and Topic can also specify the number of replicas on multiple machines. Kafka uses ZooKeeper to elect one Leader among multiple copies, and the other copies will act as Follower. Leaer is mainly responsible for reading and writing messages, that is, dealing with producers and consumers, while synchronizing messages to other copies. When there is a Broker failure, if the Topic has no Leader, a new Leader will be re-elected to solve the single point problem.

After the introduction of Broker and replica, the single point problem is solved, followed by the performance problem. For a single Topic, only the Broker where the Leader resides communicates with the producer and consumer, so the throughput will be limited by the machine on which the Broker resides. So how to improve throughput. Kafka splits the Topic into multiple partitions, that is, it divides the messages, similar to the sub-database of the database, which acts as a load balancer without the restriction of a single machine. As shown in the figure below, producer A writes messages to partitions 0 and 1 of TopicA, while consumers An and B also get messages from partitions 0 and 1. Note here that messages stored in different partitions are also different, and the concept of copy should be clearly distinguished.

From the figure above, we can see that Consumer A gets messages from partition 0 and partition 1 respectively when consuming TopicA. In order to further improve throughput, Kafka introduces the concept of consumer group, which splits consumer An into multiple consumers to form a consumer group. We can understand that Consumer An is an instance of Application A. in order to improve the throughput of consumption, we deploy several more instances of Consumer A, so that multiple consumers form a consumption group, but all they do is what Application A does and needs to be distinguished from Consumer B (different applications). In general, the number of consumers in the consumer group is the same as the number of divisions, so that a consumer can be responsible for a partition to improve efficiency. If the number of consumers in the consumption group is less than the number of partitions, there will be one consumer responsible for multiple partitions. If the number of consumers in the consumption group is greater than the number of divisions, there will be consumers who can not be divided into regions, resulting in waste. So it's generally consistent. For simplicity, and consumer group B is similar to consumer group A, consumer group B is not depicted in the following figure.

The basic concepts in Kafka are these: producers, consumers, Topic, Broker, replicas, partitions, and message groups. Finally, in order to better understand the concept of zoning, draw a detailed picture.

A partition can be regarded as a separate queue, and the producer writes messages to the partition according to the policy. There are three strategies: first, specify the partition directly; second, if no partition is specified, specify the partition through the hash function according to the key of the message; third, poll the partition if there is no key. What I want to emphasize here is that the data in the partition is different, and a message will only enter one partition. On the other hand, the consumers in the consumption group will get the corresponding messages in the partition according to the offset for consumption processing.

At this point, the study of "diagrams of basic concepts in Kafka" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.