Shulou(Shulou.com)06/02 Report--

This article focuses on "how to understand MQ Dead letter queue, retry queue, message backtracking", interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn how to understand MQ Dead letter queue, retry queue, and message backtracking.

01. Priority queue

Priority queues are different from first-in-first-out queues, high-priority messages have the privilege of being consumed first, which can provide different message levels of guarantee for downstream. However, this priority also needs to have a premise: if the consumer consumes faster than the producer, and there is no message accumulation in the message middleware server (commonly referred to as Broker), then setting the priority for the message sent is of little substantive significance, because the producer is consumed by the consumer as soon as the producer sends a message. Then there is at most one message in the Broker, and priority is meaningless for a single message.

02. Delay queue

When you shop online, do you encounter the prompt: "if you don't pay within 30 minutes, the order will be cancelled automatically"? This is a typical application scenario of delay queuing. The delay queue stores the corresponding delay message. The so-called "delay message" means that when the message is sent, the consumer does not want to get the message immediately, but waits for a specific time before the consumer can get the message for consumption. There are generally two types of delay queues: message-based delay and queue-based delay. Message-based delay means to set a different delay time for each message, so it will be re-sorted according to the delay time whenever a new message enters the queue, which will also have a great impact on performance. In practical applications, queue-based delay is mostly used, and queues with different delay levels are set, such as 5s, 10s, 30s, 1min, 5mins, 10mins and so on. The delay time of messages in each queue is the same, which avoids the performance suffering of delay sorting, and timeout messages can be delivered through a certain scanning strategy (such as timing).

03. Dead letter queue

Due to some reasons, messages cannot be delivered correctly. In order to ensure that messages will not be discarded for no reason, they are generally placed in a queue with a special role, which is generally called dead letter queue. Correspondingly, there is a concept of "fallback queue". If a consumer has an exception during consumption, then the consumption will not be Ack, and the message will always be placed at the top of the queue after the rollback operation occurs, and then it will be processed and rolled back constantly, causing the queue to fall into an endless loop. To solve this problem, you can set up a fallback queue for each queue, which, together with the dead letter queue, provides a mechanism for exception handling. In practice, the role of fallback queues can be played by dead letter queues and retry queues.

04. Retry queue

The retry queue can actually be regarded as a fallback queue, which means that when the consumer fails to consume a message, it rolls back the message to the Broker to prevent the message from being lost for no reason. Unlike the fallback queue, the retry queue is generally divided into multiple retry levels, and each retry level generally sets the redelivery delay. The more retries, the greater the delivery delay. For example, if the first consumption of a message fails to enter the retry queue Q1Magi Q1, the redelivery delay of Q1Magi Q1 is 5s, and the message is re-delivered after 5s; if the message consumption fails again, the redelivery delay of Q2Magi Q2 is 10s, and the message is delivered again after 10s. By analogy, the more times you retry, the longer it will take to re-deliver. For this reason, you need to set an upper limit. If you exceed the number of deliveries, you will join the dead letter queue. The delay level needs to be set between the retry queue and the delay queue. The difference between them is that the delay queue action is triggered internally, while the retry queue action is triggered by the external consumer. The delay queue acts once, and the scope of the retry queue is passed backward.

05. The push mode of consumption pattern push

For kafka, Broker actively pushes messages to the consumer side, which is real-time, but a certain streaming mechanism is needed to ensure that the messages pushed by the server side will not overwhelm the consumer side.

06. Consumption pattern pull mode pull

For kafka, the consumer actively requests to pull (usually timed or quantitative) messages from the broker, and the real-time performance is worse than the push mode, but the amount of messages pulled can be controlled according to its own processing power.

07. Message backtracking

A general message is processed after the consumption is completed, and then the message can no longer be consumed. Message backtracking, on the contrary, refers to the message that can be consumed before it is consumed after the consumption is completed. For messages, the problem we often face is "message loss". As to whether it is really due to the defect loss of message middleware or the misuse of the user, it is generally difficult to trace, if the message middleware itself has the function of message backtracking, we can reproduce the "lost" message by backtracking consumption and find out the source of the problem. The function of message backtracking is much more than that, for example, there are index recovery, local cache reconstruction, and some business compensation schemes can also be implemented by backtracking.

08. Message accumulation

Traffic peaking is a very important function of message middleware, and this function actually benefits from its message stacking ability. In a sense, if a message middleware does not have the ability to stack messages, it cannot be regarded as a qualified message middleware. Message stack integral memory stacking and disk stacking.

09. Message tracking / trajectory

Link tracing (trace) in distributed architecture systems is no stranger to us. For message middleware, message link tracking (hereinafter referred to as message tracking) is also important. The most popular understanding of message tracking is to know where the message comes from, where it exists, and where it is sent. Based on this function, we can carry out link tracking service for messages sent or consumed, and then we can quickly locate and troubleshoot problems. Want to know if the message was sent successfully? Why can't the messages sent be consumed on the consumer side? Why repeat consumption again? Wait a minute. By introducing the message trajectory, we can know that the message is triggered by the producer, stored by agents such as broker, and then consumed by consumers. The status, time, place and other data of each node are aggregated to form a complete link information.

10. Message filtering

Message filtering refers to the provision of specified categories of messages to downstream users in accordance with established filtering rules. In the case of kafka, it is entirely possible to send different types of messages to different topic, thus achieving message filtering in some sense, or Kafka can also classify messages in the same topic according to partition. However, in a more strict sense, message filtering should take a certain way to filter established messages according to certain filtering rules. Similarly, take Kafka as an example, messages can be filtered through the ConsumerInterceptor interface provided by the client or the filter function of Kafka Stream. For rocketmq, three modes are supported: Tag, SQL92, and class filter (new version removal).

11. Message audit

Message audit refers to the audit of the number and delay of messages in the whole process of production, storage and consumption, so as to detect whether there is data loss, whether there is data duplication, how much the end-to-end delay is, and so on. Related products: Uber Chaperone, LinkedIn kafka monitor, Confluent Control Center, etc., if you need or are interested, you can find out by yourself through the network.

twelve。 Message routing

The message is routed to the specified queue, and the consumer consumes the message in the queue. RabbitMQ can route to a specified queue or queues based on routing keys from the switch Exchanger. Kafka is routed by message topic by default. Message routing is rarely used in kafka, and it is troublesome to use. If there is no special need, it is generally not recommended.

At this point, I believe you have a deeper understanding of "how to understand MQ dead letter queue, retry queue, message backtracking". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.