Shulou(Shulou.com)06/01 Report--

RabbitMQ Shovel principle good usage is what, many novices are not very clear about this, in order to help you solve this problem, the following small series will explain in detail for everyone, there are people who need this can learn, I hope you can gain something.

Shovel reliably and consistently pulls data from a queue in one Broker (as a source) and forwards it to a switch in another Broker (as a destination). The queue as source and the switch as destination can be on the same Broker or on different brokers.

Shovel principle

Broker1 has exchanger 1 and queue1 bound by routing key "rk1"; broker2 has exchanger 2 and queue2 bound by routing key "rk2".

Configure a Shovel link between queue 1 and exchange2. When a message with the content "Shovel test payload" is sent from the client to exchange1, this message will pass through the data flow in the above figure and finally be stored in queue2. If the add-forward-headers parameter is set to true in the Shovel link configuration, there will be a special headers attribute tag when consuming this message in queue2. As shown below: The above description is the source end for the queue, the destination end for the switch, the following look at the destination end is also the queue diagram:

Although queue1 appears to forward messages directly to queue2 via Shovel link, it is actually forwarded via broker2's switch, but this switch is the default switch.

Take a look at the schematic diagram where the source end is the switch and the destination end is the switch: although it seems that the switch exchange1 directly forwards the message to exchange2 through the Shovel link, in fact, a queue (the name is customized by RabbitMQ, such as "amq. gen-ZwolUsoUchY6a 7xaPyrZZH" in the above figure) will be created in broker1 and bound to exchange1. Messages from exchange1 will be stored in this queue first, and then Shovel will pull messages from this queue and forward them to exchange2.

Exchange1, queue1, exchange2, and queue2 in broker1 and broker2 can all be created for the first time after Shovel successfully connects to the source or destination Broker (when executing a series of corresponding AMQP configuration declarations), and they do not necessarily need to be created before Shovel links are established. Shovel can configure multiple Broker addresses for the source or destination, so that after the Broker of the source or destination fails, it can try to reconnect to other brokers (random selection).

You can set the reconnect delay parameter to avoid network flooding due to reconnect behavior, or you can stop the connection directly after a failed reconnect. All configuration statements for the source and destination are re-sent after successful reconnection.

Shovel Use

Shovel plugins are also included in RabbitMQ distribution packages by default. Execute rabbitmq-plugins enable rabbitmqshovel to enable Shovel functionality. This command will also enable amqpclient plugins by default. To enable Shovel's management plug-in, execute rabbitmq-plugins enable rabbitmqshovelmanagement.

Shovel can be deployed either at the source or at the destination. Shovel can be deployed in two ways: static and dynamic. Static mode refers to setting in rabbitmq.config configuration file, while dynamic mode refers to setting through Runtime Parameter. We talk about dynamic configuration here, based on the plug-in WEB management interface for configuration.

Name: The name of the Shovel configuration

Source: In Source, you need to specify the protocol type, the source node address of the connection, and the type of the source end (queue, switch, and if it is a switch, you need to fill in routingKey)

Prefetch count: This parameter indicates the number of messages cached inside Shovel. Please refer to Federation parameters. Shovel's internal cache is the intermediate cache between the source and destination servers

Auto-delete: The default value is Never, which means that you will not delete yourself. If it is set to After initial length transferred, you will delete yourself after the message transfer is completed.

Destination: Destination needs to specify the protocol type, the destination node address of the connection, and the type of the destination (queue, switch, and if it is a switch, you need to fill in routingKey)

Add forwarding headers: If set to true, x-shovelled header attributes are added to forwarded messages

Reconnect delay: Specifies the time, in seconds, to wait before reestablishing the connection if the Shovel link fails. If set to 0, no reconnection occurs, i.e. Shovel stops working when the first connection fails. The default is 5 seconds.

Acknowledgement mode: Refer to Federation configuration. No ack means that no message acknowledgement is required;on publish means that Shovel sends each message to the destination before sending a message acknowledgement to the source; on confirm means that Shovel uses the publisher confirm mechanism to send a message acknowledgement to the source after receiving a message acknowledgement from the destination.

Shovel handles message stacking

Message pile-ups are the most common thing you encounter when working with messaging middleware. Message accumulation is a double-edged sword. A proper amount of accumulation can be used for peak clipping and caching, but if the accumulation is too serious, it may affect the use of other queues and cause the overall quality of service to decline.

The overall idea of handling message accumulation is to prepare a standby cluster for the cluster. When a queue in the online cluster has serious message accumulation, enable Shovel to import messages to one of the standby clusters. When the message accumulation in the online cluster recovers, enable Shovel to import messages from the queue in the standby cluster to the queue in the original cluster.

The following figure shows the difference between Federation/Shovel and Cluster.

Did reading the above help you? If you still want to have further understanding of related knowledge or read more related articles, please pay attention to the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.