Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "what are the functional differences between RabbitMQ and Kafka". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

I. Asynchronous message mode

Asynchronous messages can be used as a solution to decouple the production and processing of messages. When it comes to messaging systems, we usually think of two main message modes-message queuing and publish / subscribe.

1. Message queue

Message queues can be used to decouple producers and consumers. Multiple producers can send messages to the same message queue; however, when a message is processed by a messenger, the message is locked or removed on the queue and other consumers cannot process the message. In other words, a specific message can only be consumed by one consumer.

Message queue

It is important to note that if the consumer fails to process a message, the messaging system generally puts the message back in the queue so that other consumers can continue to process it. In addition to providing decoupling, message queuing can independently scale producers and consumers, and provide fault tolerance for error handling.

2. Publish / subscribe

In the publish / subscribe (pub/sub) model, a single message can be acquired and processed concurrently by multiple subscribers.

Publish / subscribe

For example, events generated in a system can be used to allow publishers to notify all subscribers. In many queuing systems, the term topics is often used to refer to publish / subscribe patterns. In RabbitMQ, a topic is a concrete implementation of the publish / subscribe model (or, more accurately, an exchange), but in this article, I will treat topics and publish / subscribe as equivalent.

In general, there are two types of subscription:

1) ephemeral subscription, which exists only when the consumer starts and runs. Once the consumer exits, the corresponding subscriptions and outstanding messages are lost.

2) persistent (durable) subscriptions, which will always exist unless deleted actively. After the consumer exits, the messaging system continues to maintain the subscription, and subsequent messages can continue to be processed.

II. RabbitMQ

As an implementation of message middleware, RabbitMQ is often used as a service bus. RabbitMQ natively supports the two message modes mentioned above. Other popular implementations of messaging middleware are ActiveMQ,ZeroMQ,Azure Service Bus and Amazon Simple Queue Service (SQS). The implementation of these message middleware has a lot in common, and many of the concepts mentioned in this article are mostly applicable to these middleware.

1. Queue

RabbitMQ supports typical out-of-the-box message queues. The developer can define a named queue, and then the publisher can send messages to the named queue. Finally, consumers can get the messages to be processed through this named queue.

2. Message exchanger

RabbitMQ uses message exchanges to implement the publish / subscribe pattern. Publishers can publish messages to a message exchange without knowing which subscribers the messages have.

Each consumer who subscribes to the switch creates a queue; the message exchange then queues the produced messages for consumption by the consumer. Message exchanges can also filter messages for some subscribers based on various routing rules.

RabbitMQ message exchanger

It is important to note that RabbitMQ supports both temporary and persistent subscription types. Consumers can call RabbitMQ's API to select the type of subscription they want.

According to the architectural design of RabbitMQ, we can also create a hybrid approach-subscribers team up and then use competition as consumers within the group to process messages on a specific queue. This group of subscribers is called a consumer group. In this way, we implement the publish / subscribe model, while also being able to scale-up subscribers to handle incoming messages.

Joint use of publish / subscribe and queues

III. Apache Kafka

Apache Kafka is not an implementation of messaging middleware. Instead, it is just a distributed streaming system.

Unlike queue-and switch-based RabbitMQ,Kafka, the storage layer is implemented using partitioned transaction logs. Kafka also provides streaming API for real-time stream processing and connector API for easier integration with various data sources; of course, these are beyond the scope of this article.

Cloud vendors offer options for the Kafka storage layer, such as Azure Event Hubsy and AWS Kinesis Data Streams. There are also specific cloud and open source solutions for Kafka streaming capabilities, but then again, they are beyond the scope of this article.

1. Theme

Kafka does not implement such a thing as queues. Accordingly, Kafka stores recordsets by category and calls this category a topic.

Kafka maintains a message partition log for each topic. Each partition consists of an ordered and immutable sequence of records, and messages are continuously appended to the tail.

When messages arrive, Kafka appends them to the end of the partition. By default, Kafka uses a polling divider (partitioner) to distribute messages consistently across multiple partitions.

Kafka can change the behavior of creating a logical flow of messages. For example, in a multi-tenant application, we can create a message flow based on the tenant ID in each message. In the IoT scenario, we can map it to a specific partition based on the producer's identity information (identity) at the constant level. Ensure that messages from the same logical flow are mapped to the same partition, which ensures that messages can be delivered to consumers in order.

Kafka producer

Consumers read out messages sequentially by maintaining partition offsets (or indexes), and then consume messages.

A single consumer can consume multiple different themes, and the number of consumers can be scaled to the maximum number of partitions available.

So when creating a topic, we need to seriously consider the expected message throughput on the created topic. The group composed of multiple consumers who consume the same theme is called the consumer group. The API provided by Kafka can handle the partition balance between multiple consumers in the same consumer group and the storage of consumers' current partition offset.

Kafka consumers

2. Message mode implemented by Kafka

The implementation of Kafka fits well with the publish / subscribe model.

Producers can send messages to a specific topic, and then multiple consumer groups can consume the same message. Each consumer group can scale independently to handle the corresponding load. Because consumers maintain their own partition offsets, they can choose persistent subscriptions or temporary subscriptions. Persistent subscriptions do not lose offsets after restart, while temporary subscriptions lose offsets after restart and start reading from the latest records in the partition after each restart.

However, this implementation scheme cannot be treated as a typical message queuing pattern. Of course, we can create a topic that is associated with a consumer group that has a consumer, so we simulate a typical message queue. However, there will be many disadvantages, which we will discuss in detail in the second part.

It is worth noting that Kafka retains messages in the partition at a preconfigured time, rather than based on whether the consumer consumes the messages. This retention mechanism allows consumers to reread previous messages freely. In addition, developers can also use Kafka's storage layer to implement functions such as event traceability and log auditing.

Although RabbitMQ and Kafka can sometimes be seen as equivalent, their implementations are very different. So we can't treat them as the same kind of tools; one is message middleware and the other is distributed streaming system.

As solution architects, we need to be able to recognize the differences between them and think as much as possible about which type of solution to use in a given scenario. The following will point out these differences and provide guidance on when and which solution to use.

4. Significant differences between RabbitMQ and Kafka.

RabbitMQ is a message broker, but Apache Kafka is a distributed streaming system. It seems that the differences can be seen in terms of semantics, but some of their internal features affect whether we can design various use cases well.

For example, Kafka is best suited for streaming data, but messages in RabbitMQ streaming are difficult to keep in order.

On the other hand, RabbitMQ has built-in retry logic and dead-letter switches, but Kafka simply leaves these implementation logic to the user.

This part mainly emphasizes their main differences between different systems.

1. Message order

RabbitMQ does not guarantee the order of messages sent to queues or switches. Although it seems logical for consumers to process messages from producers in order, this is very misleading.

There are instructions on message order assurance in the RabbitMQ documentation:

"messages published on a channel, delivered with a switch, a queue and an egress channel, will eventually be received in the order in which they are sent."

-- RabbitMQ proxy semantics (Broker Semantics)

In other words, as long as we are individual consumers, the messages we receive are orderly. However, once multiple consumers read messages from the same queue, the order in which the messages are processed cannot be guaranteed.

Because the consumer may put the message back (or retransmit) into the queue (for example, in the case of processing failure) after reading the message, the order of the message cannot be guaranteed.

Once a message is put back into the queue, another consumer can continue to process it, even if the consumer has processed the message after the message has been put back. Therefore, the consumer group processes messages out of order, as shown in the following table:

An example of losing message order using RabbitMQ

Of course, we can ensure the ordering of messages in RabbitMQ by limiting the number of concurrency of consumers to 1. More precisely, limit the number of threads in a single consumer to 1, because any parallel message processing can lead to disorder.

However, as the size of the system grows, the single-threaded consumer model can seriously affect message processing capabilities. Therefore, we should not easily choose this kind of scheme.

For Kafka, on the other hand, it provides reliable sequence assurance in terms of message processing. Kafka ensures that all messages sent to the same topic partition can be processed sequentially.

As mentioned earlier, by default, Kafka uses a circular divider (round-robin partitioner) to put messages on the appropriate partition. However, the producer can set a partition key (key) for each message to create a logical flow of data (such as messages from the same device or messages belonging to the same tenant).

All messages from the same stream are placed in the same partition so that the consumer group can process them sequentially.

However, we should also note that in the same consumer group, each partition is handled by a thread of one consumer. The result is that we cannot scale the processing power of a single partition.

In Kafka, however, we can scale the number of partitions in a topic, allowing each partition to share fewer messages, and then adding more consumers to handle additional partitions.

Winner (Winner)

Obviously, Kafka is the winner because it guarantees that messages are processed sequentially. RabbitMQ is relatively weak in this area.

2. Message routing

RabbitMQ can route messages to subscribers on a message exchange based on defined subscriber routing rules. A topic switch can route messages through a specific header called routing_key.

Alternatively, a headers switch can route messages based on any header. Both switches effectively allow consumers to set the types of messages they are interested in, thus giving solution architects a lot of flexibility.

On the other hand, Kafka does not allow consumers to filter messages in a topic before processing them. A subscribed consumer accepts all messages in a partition without exception.

As a developer, you might use Kafka streaming (job), which reads messages from topics, filters them, and finally pushes the filtered messages to another topic that consumers can subscribe to. However, this requires more work and maintenance, as well as more mobile operations.

Winner:

RabbitMQ provides better support for message routing and filtering.

3. Message timing (timing)

RabbitMQ provides a number of capabilities in determining when messages are sent to a queue:

1) message survival time (TTL)

Each message sent to RabbitMQ can be associated with a TTL attribute. The publisher can set the TTL directly or according to the policy of the queue.

The system can limit the validity of the message according to the set TTL. If the consumer does not process the message within the expected time, the message is automatically removed from the queue (and moved to the dead-letter switch, as will all subsequent messages).

TTL is especially useful for commands that are timed, because they are meaningless if they are not processed for a period of time.

2) delayed / scheduled messages

RabbitMQ can support delayed or scheduled messages through plug-ins. When the plug-in is enabled on the message exchange, the producer can send the message to the RabbitMQ, and the producer can then delay the time the RabbitMQ routes the message to the consumer queue.

This feature allows developers to dispatch future (future) commands, that is, commands that should not be processed until then. For example, when the producer encounters current-limiting rules, we may delay the execution of these specific commands to a later time.

Kafka does not provide these features. It writes messages to the partition as soon as they arrive, so that consumers can immediately get the messages for processing.

Kafka also does not use the mechanism to provide TTL for messages, but we can implement it at the application layer.

However, it is important to keep in mind that Kafka partitioning is a transaction log in append mode. Therefore, it cannot handle the message time (or the location in the partition).

Winner:

There is no doubt that RabbitMQ is the winner, because this natural implementation limits Kafka.

4. Message retention (retention)

When the consumer successfully consumes the message, RabbitMQ deletes the corresponding message from the storage. This kind of behavior cannot be modified. It is an essential part of almost any message agent design.

Instead, Kafka allocates a timeout for each topic, and messages that do not reach the timeout are retained. In terms of message retention, Kafka only treats it as a message log and does not care about the consumption status of consumers.

Consumers can consume each message indefinitely, and they can manipulate partition offsets to process these messages back and forth "in time". Kafka periodically checks the retention time of messages in the partition, and once the message exceeds the set retention period, it will be deleted.

The performance of Kafka does not depend on storage size. So, in theory, it stores messages with little performance impact (as long as your node has enough space to hold these partitions).

Winner:

Kafka was designed to save messages, but RabbitMQ is not. So this piece is not comparable, Kafka is the winner. Recommendation: the most comprehensive Java interview outline and answer analysis

5. Fault tolerant processing

When dealing with messages, queues, and events, developers often think that message processing is always successful. After all, after the producer puts each message into a queue or topic, even if the consumer fails to process the message, all it needs to do is try again until it succeeds.

Although on the face of it this method is correct, we should think more about it. First of all, we should admit that in some scenarios, message processing will fail. Therefore, even when human intervention is needed in part of the solution, we have to deal with these situations properly.

There are two possible failures in message processing:

1) transient failure-the failure is caused by a temporary problem, such as a network connection, CPU load, or service crash. We can alleviate this fault by trying again and again.

2) persistent failure-the failure is caused by a permanent problem that cannot be solved by additional retries. For example, common reasons are software bug or invalid message formats (for example, poison messages).

As architects and developers, we should ask ourselves: "how many times should we retry for message processing failures? how long should we wait between each retry? how do we distinguish between instantaneous and persistent failures?"

The most important thing is: "what are we going to do after all retries fail or encounter a lasting failure?"

Of course, different business areas have different answers, and messaging systems generally provide us with tools to implement our own solutions.

RabbitMQ will provide us with services such as delivery retry and dead letter switches (DLX) to handle message processing failures.

The main idea of DLX is to automatically send routing failure messages to DLX according to appropriate configuration information, and further process them on the switch according to rules, such as exception retry, retry count, and sending to "human intervention" queues.

Check out the following article, which provides an additional possible mode perspective on RabbitMQ processing retries.

Link: https://engineering.nanit.com/rabbitmq-retries-the-full-story-ca4cc6c5b493

The most important thing to remember in RabbitMQ is that when a consumer is processing or retrying a message (even before returning it to the queue), other consumers can process other messages after the message concurrently.

When a consumer is retrying to process a message, the message processing logic as a whole is not blocked. Therefore, a consumer can retry processing a message synchronously, no matter how long it takes to affect the operation of the entire system.

Consumer 1 is continuously retrying to process message 1, while other consumers can continue to process other messages

In contrast to RabbitMQ, Kafka does not provide this out-of-the-box mechanism. In Kafka, we need to provide and implement the message retry mechanism at the application layer.

In addition, we need to note that when a consumer is processing a particular message synchronously, other messages on the same partition cannot be processed.

Since consumers cannot change the order of messages, we cannot reject and retry a specific message and submit a message after that message. You just have to remember that partitioning is just a log in append mode.

An application layer solution can submit failed messages to a "retry topic" and process retries from that topic; but then we lose the order of the messages.

We can find an example of an Uber engineer implementation on Uber.com. If the latency of message processing is not a concern, then a Kafka scheme with adequate monitoring of errors may be sufficient.

If the consumer is blocking a retry message, the message from the bottom partition will not be processed

Winner:

RabbitMQ is the winner because it provides an out-of-the-box mechanism to solve this problem.

6. Telescopic

There are several benchmarks to check the performance of RabbitMQ and Kafka.

Although general benchmarking limits certain situations, Kafka is generally considered to have better performance than RabbitMQ.

Kafka uses sequential disk I / O to improve performance.

From the perspective of Kafka's architecture of using partitions, it will be superior to RabbitMQ in scale-out, while RabbitMQ will have more advantages in scale-up.

Large-scale deployments of Kafka can usually process hundreds of thousands of messages per second, or even millions of messages per second.

In the past, Pivotal documented an example of a Kafka cluster processing 1 million messages per second; however, it was done on a cluster of 30 nodes, and the message load was optimized to be spread across multiple queues and switches.

Link: https://content.pivotal.io/blog/rabbitmq-hits-one-million-messages-per-second-on-google-compute-engine

A typical RabbitMQ deployment consists of clusters of 3 to 7 nodes, and these clusters do not need to spread the load across different queues. These typical clusters can usually be expected to process tens of thousands of messages per second.

Winner:

Although both messaging platforms can handle large loads, Kafka is better at scaling and can achieve higher throughput than RabbitMQ, so Kafka wins this game.

However, it is worth noting that most systems have not yet reached these limits! So, unless you're building the next very popular million-user software system, you don't need to care too much about scalability, since both messaging platforms work well.

7. Consumer complexity

RabbitMQ uses intelligent agents and stupid consumer models. The consumer registers with the consumer queue, and RabbitMQ pushes the incoming message to the consumer. RabbitMQ also has pull API;, but it is rarely used.

RabbitMQ manages the distribution of messages and the removal of messages from the queue (or it may be transferred to DLX). Consumers don't need to think about this.

According to the design of the RabbitMQ structure, when the load increases, the consumer group on a queue can be effectively expanded from only one consumer to multiple consumers without any changes to the system.

Efficient scaling of RabbitMQ

Instead, Kafka uses stupid agents and smart consumer models. Consumers in the consumer group need to coordinate the theme partition lease between them (so that a specific partition is monitored by only one consumer in the consumer group).

Consumers also need to manage and store their partition offset indexes. Fortunately, Kafka SDK has been encapsulated for us, so we don't need to manage it ourselves.

In addition, when we have a low load, a single consumer needs to process and manage multiple partitions in parallel, which consumes more resources on the consumer side.

Of course, as the load increases, we just need to scale the consumer group so that the number of consumers is equal to the number of partitions in the topic. This requires us to configure Kafka to add additional partitions.

However, as the load decreases again, we cannot remove our previously added partitions, which requires more workload for consumers. Even so, as we mentioned above, Kafka SDK has done this extra work for us.

The Kafka partition cannot be removed, and consumers will do more work after scaling down

Winner:

By design, RabbitMQ is built for stupid consumers. So RabbitMQ wins this round.

5. How to choose?

Now we are faced with the multimillion-dollar question: "when to use RabbitMQ and when to use Kafka?" Summing up the above differences, it is not difficult for us to draw the following conclusions.

Criteria for giving priority to RabbitMQ selection:

Advanced and flexible routing rules

Message timing control (control message expiration or message delay)

Advanced fault-tolerant processing capabilities in situations where consumers are more likely to process messages unsuccessfully (instantaneous or persistent)

A simpler consumer implementation.

Criteria for giving priority to Kafka selection:

Strict message order

Extend message retention time, including the possibility of past message replay

High scalability that cannot be met by traditional solutions.

In most cases, these two messaging platforms can meet our requirements. However, it depends on our architects, who will choose the most appropriate tool. When making decisions, we need to consider the functional differences and non-functional limitations highlighted above.

These restrictions are as follows:

Current developers' understanding of these two messaging platforms

Availability of managed cloud solutions (if applicable)

Operating costs for each solution

The availability of SDK that applies to our target stack.

When developing complex software systems, we may be induced to use the same messaging platform to implement all the necessary messaging use cases. However, in my experience, there are usually more benefits from using these two messaging platforms at the same time.

For example, in an event-driven architectural system, we can use RabbitMQ to send commands between services and Kafka to implement business event notifications.

The reason is that event notifications are often used for event traceability, batch operations (ETL style), or audit purposes, so Kafka's message retention capability is valuable.

On the contrary, commands generally require additional processing on the consumer side, and processing can fail, so advanced fault-tolerant processing is required.

Here, RabbitMQ has a lot of highlights in terms of functionality. I may write a detailed article about it in the future, but you have to keep in mind that your mileage may change, because suitability depends on your specific needs.

This is the end of the content of "what are the functional differences between RabbitMQ and Kafka". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.