Shulou(Shulou.com)06/01 Report--

This article mainly introduces how to solve the problem of news backlog in Rabbitmq, which has a certain reference value. Interested friends can refer to it. I hope you will gain a lot after reading this article. Let's take a look at it.

Tens of millions of messages have been backlog in mq for hours and have not been solved.

1) fix the problem with consumer first, make sure it returns to consumption speed, and then stop all existing cnosumer

2) create a new topic,partition that is 10 times that of the original, and temporarily create 10 or 20 times the number of queue

3) then write a temporary consumer program that distributes data, which is deployed to consume the backlog of data

No time-consuming processing is done after consumption, and direct uniform polling is written to 10 times the number of queue temporarily established.

4) then temporarily requisition 10 times the number of machines to deploy consumer, and each batch of consumer consumes one temporary queue data

5) this is equivalent to temporarily expanding queue and consumer resources by 10 times, consuming data at a normal speed of 10 times.

6) after quickly consuming the backlog of data, we have to restore the original deployment architecture and re-use the original consumer machine to consume messages

TTL is the one that rabbitmq sets the expiration time.

If the message is overstocked in queue for more than a certain period of time, it will be cleared by rabbitmq, and the data will be gone.

Then this is the second trap. This does not mean that a large amount of data will be overstocked in mq, but that a large amount of data will be lost directly.

Solve

Lost a lot of news. We can take a plan, that is, redirect in batches, and at this time we start to write programs.

Write a temporary program to check out the lost data bit by bit, and then pour it back into mq to make up for the data lost during the day.

The difference between RAM node and disk node?

RAM node only saves metadata related to fabric (that is, RabbitMQ infrastructure such as queue, exchange, and binding) in memory, but disk node is stored in both memory and disk. The only metadata on the RAM node that is stored on disk is the address of the disk node used in the cluster. At least one disk node is required in the RabbitMQ cluster.

What is vhost? What does it do?

Vhost can be understood as virtual broker, that is, mini-RabbitMQ server. It contains independent queue, exchange and binding, etc., but the most important thing is that it has an independent authority system, which can be controlled by vhost users. Of course, from a global point of view of RabbitMQ, vhost can be used as a means of isolating different permissions (a typical example is that different applications can run in different vhost).

Thank you for reading this article carefully. I hope the article "Rabbitmq how to solve the problem of backlog of news" shared by the editor will be helpful to everyone. At the same time, I also hope you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.