Shulou(Shulou.com)06/03 Report--

This article mainly explains "how to use Kafka". The explanation in this article is simple and clear, easy to learn and understand. Please follow the ideas of Xiaobian and go deep into it slowly to study and learn "how to use Kafka" together.

1.1 defined

Kafka is a distributed Message Queue based on publish/subscribe mode, mainly used in the field of big data real-time processing.

1.2 Message Queue Application Scenarios for Traditional Message Queue

Benefits of using message queues

Decoupling allows you to extend or modify processes on both sides independently, as long as you make sure they obey the same interface constraints.

When a component of a recoverable system fails, the failure does not affect the entire system. Message queues reduce coupling between processes, so that even if a process processing a message dies, messages added to the queue can still be processed after the system recovers.

Buffering helps to control and optimize the speed at which data flows through the system, resolving situations where the processing speed of production and consumption messages is inconsistent

Flexibility & Peak Capacity Applications need to continue to function in situations where traffic surges, but such bursts are not common. It would be a huge waste to invest resources in readiness to handle such spikes. Using message queues enables critical components to withstand sudden access pressures without crashing completely due to sudden overload of requests.

Asynchronous communication Many times, users do not want or need to process messages immediately. Message queues provide asynchronous processing mechanisms that allow users to place a message in a queue without processing it immediately. Put as many messages into the queue as you want, and then process them as needed.

Two patterns of message queues

Point-to-point mode (one-to-one, consumer actively pulls data, message is cleared when received) Message producers produce messages to send to Queue, and message consumers remove and consume messages from Queue. After the message is consumed, there is no longer storage in the queue, so it is impossible for the message consumer to consume the already consumed message. Queue supports multiple consumers, but only one consumer can consume a message.

Publish/subscribe pattern (one-to-many, consumers consume data without purging messages) Message producer (publish) publishes a message to a topic, and multiple message consumers (subscribe) consume the message simultaneously. Unlike peer-to-peer, messages published to a topic are consumed by all subscribers.

Producer: message producer, that is, the client that sends messages to kafka broker;

Consumer: message consumer, client that fetches messages from kafka broker;

Consumer Group (CG): A consumer group consisting of multiple consumers. Each consumer within a consumer group is responsible for consuming data from different partitions, and a partition can only be consumed by consumers within one group; consumer groups do not affect each other. All consumers belong to a consumer group, i.e. a consumer group is logically a subscriber.

Broker: A kafka server is a broker. A cluster consists of multiple brokers. A broker can hold multiple topics.

**Topic **: Can be understood as a queue, producer and consumer oriented are both a topic;

Partition: In order to achieve scalability, a very large topic can be distributed to multiple brokers (i.e. servers), a topic can be divided into multiple partitions, each partition is an ordered queue;

Replication: To ensure that partition data on a node in the cluster is not lost when the node fails, and kafka can still continue to work, kafka provides a replica mechanism. Each partition of a topic has several replicas, a leader and several followers.

Leader: The "master" of multiple copies per partition, the object to which producers send data, and the object to which consumers consume data are all leaders.

follower: "slave" in multiple copies of each partition, synchronizing data from the leader in real time, keeping synchronization with the leader data. When a leader fails, a follower becomes a new follower.

Thank you for reading, the above is the content of "how to use Kafka". After studying this article, I believe everyone has a deeper understanding of how to use Kafka. The specific use situation still needs to be verified by practice. Here is, Xiaobian will push more articles related to knowledge points for everyone, welcome to pay attention!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.