Shulou(Shulou.com)06/01 Report--

This article shares with you about how to organize kafka knowledge points. Xiaobian thinks it is quite practical, so share it with you to learn. I hope you can gain something after reading this article. Let's not say much. Let's take a look at it together with Xiaobian.

Why do we need a messaging system?

1. Decoupling:

Allows you to extend or modify processes on both sides independently, as long as they adhere to the same interface constraints.

2. Redundancy:

Message queuing avoids the risk of data loss by persisting data until it has been fully processed. In the insert-fetch-delete paradigm used by many message queues, before removing a message from the queue, you need your processing system to explicitly indicate that the message has been processed, thus ensuring that your data is safely stored until you are finished.

3. Extensibility:

Because message queues decouple your processing, it's easy to increase the frequency of message enqueuing and processing by adding additional processing.

4. Flexibility & Peak Handling Capacity:

In the case of traffic spikes, the application still needs to continue to function, but such bursts of traffic are not common. It would be a huge waste to invest resources in readiness to handle such spikes. Using message queues enables critical components to withstand sudden access pressures without crashing completely due to sudden overload of requests.

5. Recoverability:

Failure of one component of the system does not affect the entire system. Message queues reduce coupling between processes, so that even if a process processing a message dies, messages added to the queue can still be processed after the system recovers.

6. Sequential guarantees:

In most usage scenarios, the order in which data is processed is important. Most message queues are sorted by nature and guarantee that data will be processed in a particular order. (Kafka guarantees the ordering of messages within a Partition)

7. Buffer:

It helps to control and optimize the speed of data flow through the system, and solves the situation that the processing speed of production messages and consumption messages is inconsistent.

8. Asynchronous communication:

Many times, users do not want or need to process messages immediately. Message queues provide asynchronous processing mechanisms that allow users to place a message in a queue without processing it immediately. Put as many messages into the queue as you want, and then process them as needed.

II. Kafka Architecture

2.1 topology

2.2 related concepts

1.producer： Message producers, who publish messages to terminals or services in the kafka cluster. 2.broker： Servers included in a kafka cluster. 3.topic： Each message published to the kafka cluster belongs to the category, i.e. kafka is topic-oriented. 4.partition： Partition is a physical concept, and each topic contains one or more partitions. Kafka assigns units to partitions. 5.consumer： Terminals or services that consume messages from the kafka cluster. 6.Consumer group： In the high-level consumer API, each consumer belongs to a consumer group, and each message can only be consumed by one Consumer in the consumer group, but can be consumed by multiple consumer groups. 7.replica： A copy of partition, guaranteeing high availability of partition. 8.leader： One of the roles in replica, producer and consumer, interacts only with the leader. 9.follower： A role in replica that copies data from the leader. 10.controller： One of the servers in the kafka cluster, used for leader selection and various failovers. 12.zookeeper： Kafka uses zookeeper to store meta information about clusters. 2.3 zookeeper node

kafka storage structure in zookeeper

The above is how to organize kafka knowledge points. Xiaobian believes that some knowledge points may be seen or used in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.