Shulou(Shulou.com)06/02 Report--

This article shows you how to solve the loss of RabbitMQ messages, the content is concise and easy to understand, it will definitely brighten your eyes. I hope you can gain something through the detailed introduction of this article.

What are the scenarios of RabbitMQ message loss?

First of all, let's take a look at the message cycle delivery process:

We divide the picture into three parts, left, middle and right, each of which will lead to message loss. Let's talk about how messages are lost in each stage:

1. Producer production message to RabbitMQ Server message loss scenario

1) external environment problems: the RabbitMQ Server cannot receive messages due to network packet loss and network failures, because the network in the production environment is very complex, and network jitter and packet loss are common. How to solve this problem is described below.

2) at the code level and configuration level, incomplete consideration leads to message loss.

Example 1:

In general, producers use Confirm mode to deliver messages. If the scheme is not rigorous enough, for example, if RabbitMQ Server fails to receive messages, it will send nack messages to notify producers. If producers fail to listen to messages or do nothing, there is a risk of message loss.

Example 2:

After the producer sends a message to exchange, the route sent is not bound to queue, and the message will be lost. Here are some specific examples to ensure that unexpected situations can be controlled even if they occur.

Messages stored in 2.RabbitMQ Server are missing or unreliable

1) messages are not fully persisted. When the machine is rebooted, all messages will be lost, and even Queue will be lost.

If you only persist the Message, but the Exchange,Queue does not persist, the persistence is invalid. I remember that after a buddy in the company forgot that the persistence of Queue caused the machine to restart, the Queue was gone, and naturally the Message was lost.

2) the problem of single-node mode: if a node dies, the message cannot be used, and the business may be paralyzed and can only wait.

If the message persistence scheme is made, the message will persist the hard disk, and the message will not be lost after the machine is rebooted. However, in another extreme case, the server disk suddenly breaks down (the company has encountered a lot of disk problems). Messages cannot be persisted and are not highly available, which is carefully considered in the production environment of this mode.

3) ordinary cluster mode: a node is dead, the messages on the node are not available, and the affected business is paralyzed, so you can only wait for the node to resume and restart (based on message persistence)

Although this pattern has improved a little bit, multiple nodes, but the message is still not guaranteed to be reliable, why?

Because the RabbitMQ clustering mode is a bit special, the contents of the queue only exist on one node, not all nodes, and all nodes only store message structures and metadata (which can be understood as indexes, which is also to improve performance, if there is an overhead cost to synchronize all the content to all nodes at a time). Below, I draw a picture to describe the loss of messages in a general cluster:

There are three nodes, usually one disk node and two memory nodes. First of all, the Queue1 content only exists on the node note1, which is fixed when the queue is created. Note2,note3 only stores metadata. It must be clear that Producer sends a message to note2,note2 and synchronizes the metadata to other nodes, and the content will synchronize note1.

Let's think about the Q1 problem in the figure. Note1 is down. The Queues of this node is temporarily unavailable. The node will be available after recovery.

Let's talk about the problem in Note 2 in the picture. Producer sent a message to note2,note2 that note1 hung up before synchronizing note1. How do you feel at this time. We will talk about the specific strategy later.

4) Mirror mode: it can solve the above problem, but there are still unexpected situations.

For example: when the persistent message is saved to the hard disk, the current queue node is dead, the storage node's hard disk is broken, and the message is lost. What should I do? The following will be described in more detail

3.RabbitMQ Server to consumer message lost

After the consumer receives the relevant message, the consumer machine goes down before the consumer has time to process the message. If the message is not handled properly, there will be a risk of loss. How to deal with this situation will be discussed later. The consumer also has an ack mechanism.

How to avoid message loss?

The following is also introduced from three aspects:

1. Reliability guarantee from producer production message to RabbitMQ Server

How do messages stored in 2.RabbitMQ Server guarantee

How not to lose information from 3.RabbitMQ Server to consumers?

1. Reliability guarantee from producer production message to RabbitMQ Server

In this process, messages may be lost, such as network packet loss or network failure. In general, if no measures are taken, the producer will not be able to perceive whether the message has been correctly sent to the exchange. If the producer is aware of it, it can take further processing actions, such as re-delivering the relevant message to ensure the reliability of the message.

1.1 there is usually a solution: a transaction mechanism provided by the AMQP protocol

The Channel interface in the RabbitMQ client provides several methods related to transaction mechanisms:

Channel.txSelect

Channel.txCommit

Channel.txRollback

The screenshot of the source code is as follows: public interface Channel extendsShutdownNotifier {} API in com.rabbitmq.client package

Before the producer sends the message, start a transaction through channel.txSelect, then send the message, if the message delivery server fails, roll back the channel.txRollback, then resend it, and if the server receives the message, commit the transaction channel.txCommit

However, few people do this because this is a synchronous operation, and after a message is sent, it blocks the sender to wait for a response from the RabbitMQ Server before moving on to the next message, which greatly degrades the throughput and performance of the producer production message.

Fortunately, RabbitMQ provides an improved scheme, that is, sender confirmation mechanism (publisher confirm)

First, the producer sets the channel to confirm mode by calling the channel.confirmSelect method. Once the channel enters confirm mode, all messages posted on that channel will be assigned a unique ID (starting from 1). Once the message is delivered to all matching queues, RabbitMQ will send an Basic.Ack to the producer (including the unique deliveryTag and multiple parameters of the message) This allows the producer to know that the message has correctly arrived at its destination.

In fact, the Confirm pattern can be implemented in three ways:

Serial confirm mode: every time producer sends a message, it calls the waitForConfirms () method and waits for confirm on the Broker side. If the server returns false or does not return within the timeout period, the client retransmits the message.

Batch confirm mode: every time producer sends a batch of messages, it calls the waitForConfirms () method and waits for the broker side confirm.

Asynchronous confirm mode: provides a callback method, which is called back by the producer side after one or more messages are broker confirm. Let's take a look at these three confirm modes respectively.

Serial confirm

For (int I = 0 position I)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.