Shulou(Shulou.com)06/03 Report--

If the message queue is orderly, whether to ensure that Producer and Consumer are orderly, many novices are not very clear about this, in order to help you solve this problem, the following small series will explain in detail for everyone, people who have this need can learn, I hope you can gain something.

To achieve message ordering, you need to consider both Producer and Consumer aspects.

First of all, the Producer must produce messages in an orderly manner.

Then, when consumers consume, they should also come in order, not in disorder.

Producer ordered

A common message system like RabbitMQ has a simple queue structure. Producer sends messages to the queue, and the messages entering the queue must be ordered.

Kafka is special because one of its topics (that is, the concept of queue) is actually divided into multiple partitions.

When a Producer sends a message, it is distributed among different partitions.

Producer sends messages in order, but after entering Kafka Topic, these messages may not enter which Partition, so the order must be chaotic.

If you want the messages in Topic to be globally ordered, you can only set up a Partition, which becomes a RabbitMQ structure.

However, this structure does not conform to Kafka's design concept, and Topic loses its scalability with only one Partition.

Kafka also supports a partial ordering method, which is to put a certain type of message into the same Partition, which ensures the order of the group of messages.

Specify Partition Key when sending message, Kafka performs Hash calculation on it, and decides which Partition to put according to the calculation result.

Therefore, messages like Partition Key must be together.

For example, the user ID is used as a key, so that the messages of the same user must be together, which ensures the order of this group of messages.

Consumer ordered

Messages are ordered within MQ, so Consumers are naturally received in order.

However, if multiple consumers are used, disorder may occur.

For example, a Queue of RabbitMQ has three Consumers. Although messages are received in order, their processing speeds are different, so the result is likely to be out of order.

If you want to do it strictly in order, you can only use one Consumer.

If local ordering is possible, then split the previous queue into multiple queues, just like Kafka's Partition Key, putting the same group of data into the same queue.

In Kafka, a Partition can only correspond to one Consumer, but if the Consumer uses multithreading, it will have the same effect as multiple Consumers, which will still cause disorder.

This requires further refinement of the grouping of messages.

Create a memory queue for each thread. After Consumer receives the message, it puts the messages of the same group into the same memory queue and processes them by the same thread.

To summarize, message ordering requires both Producer and Consumer ordering.

RabbitMQ queue structure is simple, the messages sent by the Producer are ordered. However, Kafka is special, a Topic has multiple partitions, if global order is required, only one Partition can be used.

If partial ordering can be accepted, you can set the Partition Key for the message, and the messages with the same Hash calculation result will be in the same Partition.

Consumer consumption needs to pay attention to the situation of multiple consumers, such as multiple consumer threads.

You can refine the grouping after the Consumer receives the message, and the messages in the same group are handed over to the same consumer thread for processing.

Did reading the above help you? If you still want to have further understanding of related knowledge or read more related articles, please pay attention to the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.