Shulou(Shulou.com)05/31 Report--

AMQP protocol and RabbitMQ principle is what, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain for you in detail, people with this need can come to learn, I hope you can gain something.

01 AMQP Protocol 1.1 Overview

AMQP: is a binary protocol with modern characteristics. Is an application layer standard high-level message queuing protocol that provides unified messaging services, is an open standard of application layer protocols, and is designed for message-oriented middleware.

1.2 Core concept

Publisher

The producer of the message. It is also a client application that sends messages to the switch Exchange.

Consumer

The consumer of the news. Represents a client application that gets a message from a message queue.

Server/Broker

Also known as Broker, accept the connection of the client and realize the AMQP entity service.

Virtual host

Virtual address, used for logical isolation, top-level message routing.

Represents a batch of switches, message queues, and related objects. There can be several Exchange and Queue in a Virtual Host, and there cannot be Exchange and Queue with the same name in the same Virtual Host.

Virtual hosts are independent server domains that share the same authentication and encryption environment. Each vhost is essentially a mini version of RabbitMQ server with its own queues, switches, bindings and permission mechanisms. Vhost is the basis of the AMQP concept and must be specified at link time. The default vhost for RabbitMQ is "/".

Message

Messages, data transferred between the server and the application. The message is anonymous and consists of Properties and Body (header and body). Properties can modify the message, such as message priority, delay, and other advanced features; Body, this is the message body content.

Exchange

The switch receives the message sent by the producer and forwards the message to the bound queue according to the routing key.

Three common switch types:

1. Direct (publish and subscribe, exactly match)

2. Fanout (broadcast)

3. Topic (topic, rule matching)

Binding

Bind. Virtual connection between Exchange and Queue, binding can contain routing key.

A binding is a routing rule that connects a switch to a message queue based on a routing key, so the switch can be understood as a routing table made up of bindings.

Routing key

Routing key. A routing rule that a virtual machine can use to determine how to route a particular message.

The queue is bound to the switch through the routing key.

When a message is sent to the MQ server, the message will have a routing key, even if it is empty. RabbitMQ also matches it with the routing key used by the binding.

If there is a match, the message will be delivered to the queue; if it does not match, the message will enter the black hole.

Connection

Connection, the application is connected to the TCP network of Broker.

Channel

Network channel is a virtual connection in TCP. Almost all operations are done in Channel, and Channel is the channel for reading and writing messages. The client can set up multiple Channel, each Channel representing a session task (similar to the session in the Connection in the database). For example, the cable is equivalent to TCP, the channel is a separate fiber bundle, and there is no problem in creating multiple channels on a TCP connection.

Once TCP is opened, an AMQP channel is created.

Whether it is publishing messages, receiving messages, or subscribing to queues, these actions are done through the channel.

Why does RabbitMQ need a channel? Why not communicate directly?

1. The creation and destruction of TCP is very expensive. It takes 3 handshakes to create and 4 breakups to destroy.

2. If the channel is not used, the application will connect to RabbitMQ with TCP. At peak times, thousands of connections per second will cause a huge waste of resources, and the number of TCP connections per second processed by the operating system is limited, which must cause performance bottlenecks.

3. The principle of channel is one thread and one channel, and multiple threads and multiple channels connect with one TCP. An TCP connection can hold unlimited channels, and even thousands of requests per second will not be a performance bottleneck.

Queue

Also known as Message Queue (consumer creation), message queues, save messages and forward them to consumers. It is the container of the message and the destination of the message. A message can be put into one or more queues. The message is always in the queue, waiting for the consumer to connect to the queue to pick it up.

02 Overview

RabbitMQ (Advanced Message Queuing Protocol, Advanced message queuing Protocol) is an open source message broker and queue server that is used to share data between completely different applications through common protocols. RabbitMQ is written in Erlang language, and RabbitMQ is based on AMQP protocol.

Reasons for RabbitMQ's high performance:

Erlang is a general concurrency-oriented programming language, which originally lies in the architecture mode of the switch field, which makes the performance of RabbitMQ through data exchange between Broker is very excellent, and Erlang has the same latency as native Socket.

03 architecture

When sending a message, you need to specify which Exchange to send, and then send it to the corresponding message queue queue with the help of routing key, and the consumer takes the message from the subscribed message queue.

As you can see from the architecture diagram, RabbitMQ is a typical producer-consumer model.

04 RabbitMQ switch

The Exchange switch receives the message and forwards the queue bound to the message according to the routing key.

Switch Properties:

Name: switch name

Type: switch type direct, topic, fanout, headers

Durability: whether persistence is required (true means persistence is required)

Auto Delete: when the last queue bound to Exchange is deleted, the Exchange is automatically deleted

Internal: whether Exchange is used internally for RabbitMQ. Default is false (this setting is not used in many scenarios).

Arguments: extension parameter, which is used to extend the custom use of AMQP protocol.

4.1 Direc Exchange

All messages sent to Direct Exchange are forwarded to the specified Queue in RouteKey.

Note: the Direct mode can use the Exchange:default Exchange that comes with RabbitMQ, so there is no need to binding the Exchange. During message delivery, the RouteKey must match exactly before it is received by the queue, otherwise the message will be discarded.

4.2 Topic Exchange

All messages sent to the Topic Exchange are forwarded to all Queue concerned with the specified Topic in the RouteKey.

Exchange makes a fuzzy match between RouteKey and a Topic, and the queue needs to bind a Topic.

Note: you can use wildcards for fuzzy matching.

4.3 Fanout Exchange

Fanout does not deal with routing keys and simply binds queues to the switch.

Messages sent to the switch are forwarded to all queues bound to the switch.

Fanout switches are the fastest to forward messages.

05 message reliability 5.1 reliability delivery

What is the reliable delivery on the production side?

1. Ensure the successful delivery of the message

2. Ensure the successful reception of MQ nodes

3. The sender receives the MQ node (Broker) acknowledgement reply ACK

4. Perfect message compensation mechanism.

5.2 solution

The solutions of Internet companies:

The message falls into the library and marks the status of the message

Delayed delivery of messages, second confirmation, callback check.

1. Message storage / persistence

Message information is stored in the repository (that is, message persistence) to mark the status of the message:

Note: this scheme requires two persistence operations on the database.

2. Delayed delivery

When messages are stored in high concurrency scenarios, the database IO is under great pressure and is not applicable. Internet companies generally use delayed delivery, secondary inspection, callback check.

Note: upstream represents the production side and downstream represents the consumer side.

1. First, the database is persisted, and then the first send message is sent

2. Send a delayed check message at the same time (check the consumption of the first message sent)

3. Consumer message

4. The consumer sends a confirmation message to Broker

5. The callback service detects the confirmation message at the consumer end and persists the state of the database (this is equivalent to an operation of the database and asynchronous persistence)

6. The callback service responds to the second delayed message to confirm that the message is consumed successfully. If an exception occurs, the callback service invokes RPC to the producer and sends it again.

06 message idempotency 6.1 idempotent

Idempotency means that the data is manipulated several times and is still guaranteed to be correct.

Idempotency on the consumer side means that our messages will never be consumed multiple times, even if we receive more than one message.

6.2 solution

The industry's mainstream idempotent operation:

Unique ID+ fingerprint code mechanism

Using the atomicity of Redis

1. Unique ID+ fingerprint code mechanism

Using a unique ID does not guarantee uniqueness (users may perform multiple consumption in a short period of time), and a fingerprint code is required (this may be generated by business rules, such as timestamps, or the database primary key).

Perform SQL:

SELECTCOUNT (1) FROMT_ORDERWHEREID= unique ID+ fingerprint code

The ISNERT operation can be performed if 0 is returned, but not if 1 is returned.

Advantages:

Easy to implement

Disadvantages:

Performance bottleneck of database writes under high concurrency

Solution:

Follow up the ID to divide the database and table for algorithm routing

2. Using the atomicity of Redis to realize

It is realized by the atomic operation of Redis itself. Using Redis for idempotency, you need to consider:

1. Are we going to store the data in the database? if so, the key problem to be solved is how to achieve atomicity in the database and cache?

2. If it is not implemented, it will be stored in the cache. How to set the policy of timing synchronization?

07 message delivery 7.1 Confirm confirmation message

The confirmation of the message means that after the producer delivers the message, if the Broker receives the message, it will give us a reply to the producer.

The producer receives the reply to determine whether the message is normally sent to Broker, which is also the core guarantee of the reliable delivery of the message!

How do I implement Confirm confirmation messages?

1. Enable the confirmation mode on channel: channel.confirmSelect ()

2. Add a listening addConfirmListener to the channel, listen for the returned results of success and failure, resend the message according to the specific results, or log and other subsequent processing.

7.2 Return return message

Return Listener is used to handle non-routable messages!

Our message producer sends the message to a queue by specifying an Exchange and RoutingKey, and then our consumer listens to the queue for consumption processing.

But in some cases, if the current Exchange does not exist or the specified route key route is not available when we send the message, we need to use Return Listener if we need to listen for this kind of unreachable message.

08 message side current limit

Suppose such a scenario, first of all, there are tens of thousands of outstanding messages on the RabbitMQ server. If we open a consumer client at random, the following will happen: a huge amount of messages will be pushed in an instant, but our single client cannot process so much data at the same time.

RabbitMQ provides a QOS (quality of service) function, that is, under the premise of non-automatic confirmation of messages, if a certain number of messages (through setting the value of Qos based on consumer or channel) are not confirmed, new messages are not consumed.

09 message side confirmation

When the consumer side carries on the consumption, if due to the business anomaly, we can record the log, and then make compensation. If there are serious problems such as server downtime, then we need to manually ACK to ensure the success of consumer consumption.

Message acknowledgement ACK: if the consumer's server has an exception while processing the message, then maybe the message being processed does not complete the message consumption and the data will be lost. To ensure that data is not lost, RabbitMQ supports message confirmation ACK.

ACK's message confirmation mechanism is that consumers receive a message from RabbitMQ and process it, and then send it back to RabbitMQ,RabbitMQ to receive feedback before deleting the message from the queue.

1. If a consumer has network instability or server exception when dealing with a message, then there will be no ACK feedback. RabbitMQ will think that the message is not consumed normally and will put the message back into the queue.

2. If you are in the case of a cluster: RabbitMQ will immediately push this message to other consumers online. This mechanism ensures that no messages and tasks are lost in the event of a consumer server failure.

3. Messages are never deleted from RabbitMQ: messages are deleted from RabbitMQ server data only when consumers send ACK feedback correctly and RabbitMQ acknowledges receipt.

4. The ACK confirmation mechanism for messages is turned on by default.

Considerations for the development of ACK mechanism:

If you forget ACK, the consequences will be serious. When Consumer exits, Message will always be redistributed, and then RabbitMQ will take up more and more memory, and this "memory leak" is fatal because RabbitMQ runs for a long time.

10 return to queue / TTL queue / Dead letter queue 10.1 return to queue mechanism

The consumer returns to the queue in order to re-deliver the message to the Broker for those messages that have not been successfully processed.

Generally, in practical applications, it will be closed and returned to the queue, that is, it will be set to false.

10.2 TTL queues / messages

TTL is an acronym for Time To Live, which means time to live.

RabbitMQ supports the expiration time of messages, which can be specified when the message is sent.

RabbitMQ supports the expiration time of the queue, which is calculated from the time the message is queued. As long as the timeout configuration of the queue is exceeded, the message will be cleared automatically.

10.3 Dead letter queue

Dead letter queue (DLX,Dead-Letter-Exchange). With DLX, when a message becomes a dead message in one queue, it can be re-publish to another Exchange, which is DLX.

There are several situations where a message becomes a dead letter:

1. Message is rejected (basic.reject/basic.nack) and request=false

2. Message TTL expires

3. The queue reaches the maximum length.

DLX is also a normal Exchange, no different from a normal Exchange, it can be specified on any queue, which is actually setting the properties of a queue.

When there is a dead letter in the queue, RabbitMQ automatically redistributes the message to the set Exchange and is routed to another queue.

You can listen to the messages in this queue for processing, which can make up for the previous immediate parameter function of RabbitMQ3.0.

Settings for the Dead letter queue:

1. First, you need to set the Exchange and queue of the dead letter queue, and then bind them:

Exchange:dlx.exchangeQueue:dlx.queueRoutingKey:#

2. Then, we declare the switch, queue, and binding normally, but we just need one parameter on the queue machine: arguments.put ("x-dead-letter-exchange", "dlx.exchange"); so that when the message expires, requeue, and the queue reaches its maximum length, the message can be routed directly to the dead letter queue.

11 RabbitMQ common commands

Rabbitmqctl stop_app: close the application

Rabbitmqctl start_app: launch the application

Rabbitmqctl status: node statu

Rabbitmqctl add_user username password: adding user

Rabbitmqctl list_users: list all users

Rabbitmqctl delete_user username: deleting a user

Rabbitmqctl clear_permissions-p vhostpath username: clear user permissions

Rabbitmqctl list_user_permissions username: list user permissions

Rabbitmqctl change_password username newpassword: changing password

Rabbitmqctlset_permission-p vhostpath username ". *": set user permissions

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.