Shulou(Shulou.com)06/01 Report--

Rabbitmq how to ensure not to lose news, many novices are not very clear, in order to help you solve this problem, the following editor will explain for you in detail, people with this need can come to learn, I hope you can gain something.

Background introduction:

The author recently studied rabbitmq and wondered how it guaranteed not to lose information. So I sorted out this article to share with you, my own understanding, if there are inaccuracies or different opinions, please also give feedback, we can discuss, learn from each other and grow from each other.

Basics:

Before I start to discuss this issue, I still feel it is necessary to review the basic knowledge such as the architecture of rabbitmq, as follows:

For services that use rabbitmq, they are mainly composed of three parts: producer, rabbitmq, and consumer. These three communicate through the network, in which the producer corresponds to the exchange, the consumer corresponds to the connection, and the rabbitmq is composed of three parts of exchange,queue,connection.

Message flow: after the message is produced by the producer, the message is reported to exchange,exchange bound and stored in queue, and then delivered to the final consumer.

In this way, the whole process is divided into three scenarios:

Scenario 1: how to ensure that the reported messages of producers and exchange will not be lost?

For network communication, the best way to solve the problem of data loss is the message confirmation mechanism, which is guaranteed in two ways in rabbitmq: one is the transaction mechanism, which is guaranteed at the amqp protocol level. The specific operations are as follows:

There are three methods related to transaction mechanism in RabbitMQ: txSelect (), txCommit () and txRollback (). TxSelect is used to set the current channel to transaction mode, txCommit is used to commit the transaction, and txRollback is used to roll back the transaction. After starting the transaction through txSelect, we can publish the message to the broker proxy server. If the txCommit commit is successful, the message must have reached broker. If the broker exception crashes or for some other reason throws an exception before the txCommit executes, we can catch the exception and roll back the transaction through the txRollback. (note: using transactional implementation will reduce the message throughput of RabbitMQ.)

The steps are:

1. ClientMurray-send-> Tx.Select2.broker---- send-> Tx.Select-Ok (after publish) 3.client-send-> Tx.Commit4.broker- send-> Tx.Commit-Ok

One is the confrim mechanism:

Principle: message response mechanism

The producer sets the channel to confirm mode. Once the channel enters confirm mode, all messages posted on that channel will be assigned a unique ID (starting from 1). Once the message is delivered to all matching queues, broker will send an acknowledgement to the producer (the only ID containing the message), which allows the producer to know that the message has reached the destination queue correctly. If the message and queue are persistent, the acknowledgement message is written to disk and sent out, and the deliver-tag field of the acknowledgement message returned by broker to the producer contains the sequence number of the acknowledgement message.

The advantage of confrim is that it is asynchronous. After the producer has sent a message, it is not necessary to wait for the message to return, then it can continue to process another message and wait for the message's ack to return before processing the previously sent message, which is similar to the practice of multiplexing. After receiving it, rabbitmq will reply to ack, and if it is caused by a problem with rabbitmq itself, it will reply to nack.

For the producer, in order to confirm whether the message has really reached the rabbitmq side, it is also necessary to set a timeout resend on the producer side. After all, the message may be lost in the network.

API used in confrim mode:

Https://godoc.org/github.com/streadway/amqp#Channel.Confirm

Scenario 2: how can consumers get messages from queue to ensure that they are not lost?

After the message arrives at rabbitmq, it saves the data to queue. Queue is stored in memory, but rabbitmq provides persistence operation. This strategy is as follows:

The 1.buffer is about 1m. When it is full, it will be written to disk. The 2.25ms timeout. If buffer is not satisfied, the timeout will be written to disk.

Still, it's possible to lose data, especially when rabbitmq dies when buffer doesn't write to disk. However, rabbitmq also provides a way to mirror the queue, using the active / standby method to prevent messages from being lost. However, when master and salve die at the same time, the data will still be lost, but the probability of dying at the same time will be much less. (the author thinks that there is no 100% loss of information, but the probability of losing information becomes very low. )

Reference article: https://blog.csdn.net/u013256816/article/details/60875666

Scenario 3: how to ensure that messages are not lost within rabbitmq?

For consumers, message response is also used to prevent messages from being lost, but the ack mechanism is used in this layer, but the ack can be set to not waiting for ack and waiting for ack. Here we use setting ack.

Consumers usually respond to messages in the following three scenarios:

1. After receiving the rabbitmq message, and after processing the message, the consumer will actively reply to the ack message. After receiving the ack, rabbitmq will continue to assign the next message to the consumer for processing.

two。 Of course, the rabbitmq can also reply to the unack message, so that the message queue will continue to assign the message to the consumer next time for message reprocessing.

3. The consumer first replies back the ack message, and then re-puts the message into the rabbitmq, so it is realized through the queue feature of rabbitmq. The message retry, the retry here, is not to process this message all the time, but to wait until the messages in the queue are queued to it.

In addition, rabbitmq also implements the concept of batch, through qos to set the maximum number of messages allocated to a consumer at one time, allowing consumers to consume a batch of messages at a time, and then assign the next batch of messages to this consumer after processing these messages.

Question1: once the consumer does not reply to Ack messages for a long time or the consumer is stuck, how to deal with this scenario?

In theory, consumers need to implement a timeout handling mechanism, which is not finished within a certain period of time, and need to time out to reply ack or unack messages to rabbitmq.

Question2: even if the consumer has a timeout mechanism, what will happen if the message is lost when the consumer sends ack to rabbitmq, and the rabbitmq message does not receive a response message?

Rabbitmq provides a processing mechanism for consumers to hang up, which will be detected after consumers hang up, and then processed later.

Rabbitmq also has a ttl function that sets the corresponding ttl value for a message queue or a single message. once the ttl times out, the message will become dead message and will no longer be assigned to consumers. We can use this strategy here. After the message becomes a dead message, we can ask the producer to produce the same message again and put it in the rabbitmq. If it is determined that the message is no longer in use, we can simply discard the message.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.