Shulou(Shulou.com)06/01 Report--

This article will explain in detail how RabbitMQ can efficiently deploy distributed message queues. The editor thinks it is very practical, so I share it with you as a reference. I hope you can get something after reading this article.

First, the sky falls a strange soldier

1. Message queuing uses messages to connect applications. These messages are routed between applications through a message proxy server such as RabbitMQ

2.RabbitMQ advantages:

Apart from Qpid, the only proxy server that implements the AMQP standard

Because Erlang,RabbitMQ clusters are incredibly simple

More reliable than others and better able to prevent collapse

3. Check server status: rabbitmqctl status

Second, understand message communication

a. Consumers and producers

1.RabbitMQ is not a fast food car but a message delivery service that acts as a router between the application and the server.

two。 The producer (producer) creates the message and then publishes (sends) it to the proxy server (RabbitMQ). Message contains payload (payload) and label (label)

The payload is the data you want to transmit, which can be anything.

The tag describes the payload and uses it to determine who will get a copy of the message

3. The messenger, who connects to the proxy server and subscribes to the queue, gets only part of the message: the payload when the consumer receives the message

4. Producers create messages and consumers receive them. Applications can act as producers to send messages to other applications. Or as a consumer, receive messages. You can also switch between the two. However, a channel (channel) must be established first.

5. Create a TCP connection between the application and the Rabbit proxy server. Once the TCP connection is found (authenticated), the application can create an AMQP channel. Channels are virtual connections established within a "real" TCP connection, and each channel is assigned a unique ID

b. Build from the bottom: queu

1.AMQP message routing must have three parts: switch, queue, and binding

The producer publishes the message to the switch; the message eventually arrives in the queue and is received by the consumer; the binding determines how the message is routed from the router to a specific queue

two。 Queues are like named mailboxes, where messages eventually reach the queue and wait for consumption, and consumers receive messages from specific queues in two ways:

Subscribe through AMQP's basic.consume command: this command should be used if consumers process queue messages and / or need to receive messages automatically as soon as they arrive on the queue

Basic.get command: subscribe to a message, get a single message, and then unsubscribe. Do not put it in a loop instead of basic.consume, which will affect performance.

3. When the Rabbit queue has multiple consumers, messages received by the queue will be sent to the consumer in a round-robin, with each message sent to only one subscribed consumer

4. Consumers must either send an acknowledgement to RabbitMQ explicitly through AMQP's basic.ack, or set the auto_ack parameter to true when subscribing to the queue. The consumer's confirmation of the message has nothing to do with telling the producer that the message has been received. The consumer tells RabbitMQ that it has received the message correctly through the confirmation command, and RabbitMQ can safely delete the message from the queue.

5. If the consumer receives a message and confirms that it was previously disconnected from Rabbit (or unsubscribed from the queue), RabbitMQ will assume that the message is not distributed and then redistribute it to the next subscribed consumer

6. Reject message:

Disconnect consumers from the RabbitMQ server: this will cause RabbitMQ to automatically re-queue and send messages to another consumer, but the disadvantage is that the way you connect / disconnect adds to the burden on RabbitMQ

Versions above RabbitMQ2.0.0 can use AMQP's basic.reject command

7. Both consumers and producers can use AMQP's queue.declare command to create queues

If consumers subscribe to another queue on the same channel, they can no longer declare the queue. They must first unsubscribe and set the channel to "transport" mode.

Consumers need a queue name when they subscribe to a queue, and also specify a queue name when creating a binding

Exclusive: if set to true, the queue becomes private and only your own application can consume the queue

Auto-delete: when the last consumer unsubscribes, the queue is automatically removed

Passive: when set to true, queue.declare will return successfully if the queue exists; if it does not exist, an error will be returned

8. If you can't afford to lose a message into a "black hole", both producers and consumers should try to create queues; otherwise, consumers can only declare queues.

c. Join together: switches and bindings

1. When you send a message to the proxy server, the message will have a routing key-even if it is empty-and RabbitMQ will match it with the routing key used by the binding. If there is a match, the message will be delivered to the queue. If no binding mode is matched, the message will enter the "black hole"

two。 Switches and bindings can complete different usage scenarios; for publishers who send messages to the server, there is no need to care about the logic of the other side of the server (queues and consumers in the entire message processing process)

3. Protocols that handle delivery to multiple queues:

Direct switch: if the routing keys match, the message is delivered to the corresponding queue. The server must implement a direct type exchange that contains a default exchange with a blank string name. When a queue is declared, it is automatically bound to the default switch and uses the chat table name as the routing key $channel- > basic_publish (message content, default switch, routing key)

Fanout switch: the received message is broadcast to the bound queue, and when a message is sent to the fanout switch, the message is delivered to all queues attached to this switch

Topic switch: enables messages from different sources to reach the same queue

d. Multi-tenant mode: virtual hosting and isolation

1. Each RabbitMQ server can create a virtual message server, called a virtual machine (vhost). Each vhost is essentially a mini version of RabbitMQ server, with its own queues, switches and bindings, as well as permission mechanisms.

two。 When creating a user in Rabbit, the user is usually assigned to at least one vhost and can only access queues, switches, and bindings within the assigned vhost. When designing the message communication architecture, keep in mind that the vhost is absolutely isolated, and when the vhost is created on the cluster, the vhost is created throughout the cluster.

3. Create, delete, view:

Rabbitmqctl add_vhost [vhost_name]

Rabbitmqctl delete_vhost [vhost_name]

Rabbitmqctl list_vhosts

e. Persistence and strategy

1. The durable property of queues and switches is set to true, which determines that RabbitMQ recreates the queue (or switch) after a crash or restart.

two。 For a message to recover from a Rabbit crash, you must:

Set its delivery mode (delivery mode) option to 2 (persistent)

Send to persistent switch

Arrive at the persisted queue

3. The way to ensure that persistent messages can be recovered is to write them to a persistent log file on disk. When a persistent message is published to a persistent switch, Rabbit will not send a response until the message is submitted to the log file. After that, if the message is routed to a non-persistent queue, it will automatically remove it from the persistent log.

4. Disadvantages of persistence: slower performance and poor performance in built-in clusters

5. Use persistence mechanism for critical messages

6.AMQP transactions will greatly degrade the performance of Rabbit. The Rabbit team uses sender acknowledgment mode and sets the channel to confirm mode, which is asynchronous.

Https://github.com/zhangyue0503/rabbitmq/tree/master/2

III. Running and managing Rabbit

a. Server management

1. Node: describes an Erlang node running an Erlang application. Multiple applications can run on the same node. RabbitMQ node refers to the RabbitMQ application and its Erlang node.

2.rabbitmq-server startup; rabbitmqctl stop shutdown

3. The configuration file rabbitmq-config,vm_memory_high_watermark specifies the memory that can be consumed, with a decimal number of 0.4 representing 40%

b. Request permission

1. User related commands:

Rabbitmqctl add_user username password

Rabbitmqctl delete_user user name

Rabbitmqctl list_users

Rabbitmqctl change_password username New password

two。 Composition of access control entries: user granted access, vhost of permission control application, combination of read / write / configuration permissions to be granted, permission scope

3. Access control entries cannot span vhost

4. Access command:

Rabbitmqctl set_permissions-p [vhost] [user] "configure", "write", "read", ".*" match any queue and switch, "check-.*" matches queues and switches that begin with check-, "" does not match queues and switches

Rabbitmqctl list_permissions-p [vhost] verify that permissions are correct

Rabbitmqctl clear_permissions-p [vhost] [user] removes the user's permissions on vhost

Rabbitmqctl list_user_permissions [user] View the user's permissions on all vhost

c. Check

1. Related commands:

Rabbitmqctl list_queues-p [vhost] [name name, number of messages messages, number of consumers consumers, memory memory usage, etc.] List all queues and the number of messages in the queue

Rabbitmqctl list_exchanges view switch information

Rabbitmqctl list_bindings to view binding information

two。 Journal

*-sasl.log,SASL (system Application support Library) is a collection of libraries that provide a series of standards to help develop Erlang applications

Rotation log: rabbitmqctl rotate_logs shuffix,shuffix is usually a number, added to the end of the rotated log file

d. Fix Rabbit: troubleshoot

1.RabbitMQ uses Mnesia database to store queues, switches, bindings, and so on.

Https://github.com/zhangyue0503/rabbitmq/tree/master/3

Fourth, solve the problems related to Rabbit: coding and mode

a. Decoupling

1. Asynchronous state thinking (separating requests and actions): receive requests-> RabbitMQ- > process requests (storage)

two。 Automatic polling (round-robin) can replace load balancing

3. Zero-cost API: language should not be a yoke

4. How to split the application? Which part of the application is the order recipient and which part is the order processor?

b. Forget the model immediately after sending it

1. Create the task, put it on the switch, and get your application back to work

two。 There are two general types of tasks that match the pattern:

Batch processing: work or transformation for large data collections

Notification (notifications): to describe the occurrence of the event

3.RabbitMQ uses "." As a delimiter for different parts of the tag; no_ack=false tells RabbitMQ to display the acknowledgement message, which pauses sending new messages from the queue until the last message received is processed and an acknowledgment message is sent

c. Implement RPC with RabbitMQ and wait for a response

1. Use reply_to as the destination for publishing reply messages, and you do not need to specify a switch when publishing

2.exclusive=true, make sure that only you can read the data on the queue; auto_delete, when you disconnect the queue after receiving the message, Rabbit will automatically delete the queue

Https://github.com/zhangyue0503/rabbitmq/tree/master/4

5. Cluster and processing failure

a. Cluster architecture

1. The cluster creates complete queue information (metadata, status, content) only on a single node, not on all nodes.

two。 The switch is just a binding list of a name and a queue, and the channel is the real router.

3. Using AMQP transactions, messages are blocked until they are routed to the queue, or the sender acknowledgment (publisher confirm) mode is used to record messages that have not been acknowledged when the connection is broken. These two solutions can help detect messages that cannot be routed when the node fails and the destination queue no longer exists

4. A single node must be a disk type node, otherwise all configuration information will be lost when rebooted; the cluster allows only one node to be a disk node, the other can be a memory node, and cannot operate on the queue when the disk node crashes; two disk nodes can be set up; when adding internal nodes, you need to make sure that all disk nodes are informed

b. Stand-alone cluster

1. You need to modify the environment variables RABBITMQ_NODE_PORT and RABBITMQ_NODENAME, such as RABBITMQ_NODE_PORT=5763 and RABBITMQ_NODENAME=rabbit_1, and then rabbitmq-server-detached

2.rabbitmqctl-n rabbit_1@ hostname stop_app, then reset

3.rabbitmqctl-n rabbit_1@ hostname join_cluster rabbit@ hostname

4.rabbitmqctl-n rabbit_1@ hostname start_app

5.rabbitmqctl cluster_status

6. Murn indicates that commands are executed on the specified node rather than the default node

c. Distribute nodes to more machines

1. You need to copy to find .erlang.cookie, copy the string to another node, and then join_cluster it.

two。 Delete a node directly through the reset command

d. Mirror queues and reserved messages

1. The master copy of the mirror queue exists on only one node (master queue, master) and has a slave queue (slave) copy on the other cluster

2.xa-ha-policy and xa-ha-policy-params

Https://github.com/zhangyue0503/rabbitmq/tree/master/5

6. Recover from failure

a. Load balancing for Rabbit

1. When adding a load balancer to the Rabbit, the cluster node acts as the server behind the load balancer, and your producers and consumers are the customers. The application only needs to know that the IP; load balancer at the front end of the load balancer transparently connects the client to the cluster node with minimum connection load.

b. Connection loss and failover

1. Failover should always be treated as a connection to a completely unrelated RabbitMQ server rather than a cluster node with a shared state. Whenever a node failure occurs, the first task after the failure is detected and repeated is to construct switches, queues, and bindings.

7. Warren and Shovel: failover and replication

A.warren: another way of clustering

1. A warren is a pair of primary / standby independent servers with a load balancer in front of it to handle failover. This is a true no-sharing architecture, and there is no collaboration between the master and the backup.

b. Long-distance communication and replication

1.Shovel is a plug-in for RabbitMQ that defines the relationship between a queue on RabbitMQ and a switch on another RabbitMQ

two。 Install the plug-in: rabbitmq-plugins enable rabbitmq_shovel

Manage RabbitMQ from the web side

a. Beyond the rabbitmqctl:RabbitMQ Management plug-in

1.rabbitmq-plugins enable rabbitmq_management, http://localhost:15672/mgmt/

2.rabbitmqadmin script: wget http://localhost:15672/cli/rabbitmqadmin

Use REST API to control Rabbit

1. Http://localhost:15672/api, such as http://localhost:15672/api/users

X. Monitoring

1. There are four kinds of integer exit codes: 0murOKJEN 1MWARNINGJEI CRITICALPKI 3RAPHANOKNOWN

Elevating performance and ensuring safety

1. If you can accept missing messages, set delivery-mode to 1 and 2 to persist, and messages are written to disk, slowing down

2.no-ack is true, and the server will automatically dequeue the message after it is sent to the client.

The difference between the 3.direct switch and the fanout switch is that the latter ignores the routing key when querying the rabbit_route table; the topic switch takes up more memory than the former two

4. If the mandatory and immediate in the message are marked as false, the message will be delivered asynchronously

5.RabbitMQ is optimized to deliver messages to consumers as quickly as possible and should keep the queue empty as much as possible

6. If durable is true in the queue declaration, queue records will be added to the tables rabbit_queue and rabbit_durable_queue, otherwise the records will only be stored in rabbit_queue

XII. Expand RabbitMQ

1. Http://www.rabbitmq.com/plugins.html

two。 Plug-in enable: rabbitmq-plugins enable xxxxx; remove plug-in: rabbitmq-plugins disable xxx

This is the end of the article on "how to efficiently deploy distributed message queues in RabbitMQ". I hope the above content can be helpful to you, so that you can learn more knowledge. if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.