Shulou(Shulou.com)06/01 Report--

This article mainly shows you "what the architecture is like in rabbitMq". The content is simple and clear. I hope it can help you solve your doubts. Let the editor lead you to study and learn this article "what the architecture is like in rabbitMq".

One: characteristics

Advantages: 1. Lightweight and easy to deploy. 2. Message routing rules are flexible. 3. The message delay is at the microsecond level.

Disadvantages: 1, the performance is relatively weak, the processing capacity of a single machine is in the order of 1W per second. 2. If a large number of messages accumulate, the performance will be greatly degraded. 3. The possibility of losing information is very low, though very low.

Therefore, if the system order of magnitude is not very large, tens of thousands-hundreds of thousands of throughput, there is no very high performance requirements, the choice of rabbitmq is still very good, convenient and flexible to use, rich routing, low delay, good reliability, even if there is a very small possibility of data loss, you can also use some methods to track (later on).

Second: logical architecture

Message delivery process:

When the switch Exchange is bound to the queue Queue, a binding key bindingKey is specified. When the producer sends a message, a routing key routingKey is specified in the message. After the message arrives at Exchange, the message is routed to the appropriate Queue compared with bingdingKey and routingKey. Consumers can consume messages in the queue through the mode of actively pulling or listening. (of course, not all messages go through the switch, which will be discussed below)

Three: work flow

The producer sends a message:

The producer connects to RabbitMQ, establishes TCP connection (Connection), and opens channel (Channel).

The producer declares an Exchange (switch) and sets relevant properties, such as switch type, persistence, etc.

The producer declares that a queue well sets relevant properties, such as whether it is exclusive, whether it is persistent, whether it is automatically deleted, and so on.

Producers bind switches to queues (binding) through routingKey (routing key)

The producer sends a message to RabbitMQ Broker containing information such as routingKey (routing key), switch, etc.

The corresponding switch looks for a matching queue according to the received routingKey.

If found, messages sent from the producer are stored in the appropriate queue.

If it is not found, it will be discarded or returned to the producer according to the attributes configured by the producer.

Close the channel.

Close the connection.

Consumers receive messages:

Consumers connect to the RabbitMQ Broker, establish a connection (Connection), and open a channel (Channel).

Consumers request from RabbitMQ Broker to consume messages in the corresponding queue, and the corresponding callback function may be set, and

Do some preparatory work

Wait for the RabbitMQ Broker to respond and deliver the message in the corresponding queue, and the consumer receives the message.

The message received by the consumer confirmation (ack).

RabbitMQ removes the corresponding acknowledged message from the queue.

Close the channel.

Close the connection.

Relationship between Connection and Channel:

Producers and consumers need to establish a tcp connection, or connection, to broker (an instance of mq). Then an AMQP channel channel is created based on connection, which is actually a virtual connection, and all instructions are done through channel.

Why use channel, because multiple channel can reuse the same connection to reduce performance overhead and make it easy to manage (otherwise, if there are multiple threads sending and receiving messages, frequent TCP connections are created and destroyed). However, if the channel traffic is too large, you need to establish multiple connection to share.

Four: switch type

Premise: bingdingKey and routingKey are a string with "." To separate, such as user.news

There are four common types of switches: fanout, direct, topic and headers.

Fanout: this type of switch routes messages to all queues bound to it, no matter what key you are.

Direct: queues that match bingdingKey and routingKey receive messages.

Topic: an upgraded version of direct that can use wildcards for bingdingKey. There are two kinds of wildcards: # and *, where * is used to match one word, and "#" is used to match 0 to more words. For example, bingdingKey is: user.#. Then in the message sent, the first word of routingKey is user. BingdingKey is: user.*.name. Then the first and third words of routingKey must be user and name, and can only be three words.

Headers: headers this switch is amazing, because the performance is very poor, generally not very useful, but we still need to know. The headers header switch, which does not need bingdingKey and routingKey, but relies on the property value set in the header, is a map. When binding the switch and the queue, you also need to specify a header attribute, which is also a map. It is worth noting that a key-value pair put is required in the map when binding: map.put ("x-match", "all") or map.put ("x-match", "any"). The difference is: all: all key and value of the map in the message, all of the bound map should be able to match. Any: as long as there is a key-value pair that matches.

In fact, headers and direct are very similar, but you can support multiple properties at the same time, and the type is not just a string.

Five: data storage mechanism there are two types of RabbitMQ messages: persistent messages are written to disk when they arrive in the queue, and a backup is saved in memory at the same time. When memory is tight, messages are cleared from memory. Memory is designed to improve performance. Non-persistent messages: generally only stored in memory, when the memory pressure when the data flush disk processing, in order to save memory space.

The RabbitMQ storage layer consists of two parts: queue indexing and message storage.

As shown above:

If you send a non-persistent message, the index will be maintained in index file, and the message will be stored in the store file in memory. If there is not enough memory in the message stack, the message will be brushed to the msg_store_transient file on disk (of course, the maintenance of the index will also be flushed).

Send persistent messages, first save to disk msg_store_persistent, and store the performance improvement in memory stroe file.

Queue index: rabbit_queue_index (private to each queue) maintains message information such as storage location (the specific location of the message in the message storage below), whether it has been received by consumers, confirmation of consumption, etc. File form: use sequential segment files with the suffix .idx, the file name starts at 0, and each file has N records (N = the value of the segment_entry_count parameter, default 16384). When each index reads messages from disk, at least one segment file is maintained in memory. Why at least? Because the segment file in which the index is located will be loaded into memory, and if the message size is small, it will also be directly stored in the index file, so there may already be a needed message in the loaded segment file. So, how small messages will be put directly into the index file? Determined by the parameter queue_index_embed_msgs_below, the default is 4096B. This value must be set carefully, even if the setting is a little larger, the memory may explode.

Message store: rabbit_msg_store (shared by all queues under a virtual machine)

Storage: storage in the form of key-value pairs. Rabbit_msg_store is divided into persistent msg_store_transient and non-persistent msg_store_persistent, which is lost after the non-persistent restart service. The file is suffixed with .rdq, the file name starts at 0, and a new file is created when the size of the file_size_limit is exceeded, and the message is written as an append. The location mapping and file information of the message in the file are recorded in the ETS (Erlang Term Storage) table when the message is stored.

Read: according to msg_id to find the storage file, if it exists and is not locked, directly open the file to read from the specified location, send the request by rabbit_msg_store, let it handle itself.

Delete: after the message is consumed, it is not deleted directly, but marked as junk data, and then deleted later. The steps are as follows: 1. Delete the record in the ETS table and update the relevant information in the corresponding file, such as consumption in the index file and junk data in the store file. 2. When a file is full of junk data, you can delete the file. But when will the recycling delete be triggered? 3. When two logically adjacent files are detected, their valid data can be merged into one file, and when the size of the garbage data exceeds n times the total file data size (at least 3 files) (n is the value of the parameter GARBAGE_FACTORION, the default is 0.5), the merge and garbage collection are triggered. 4, when merging, first lock two files, sort out the valid data of the two files respectively, merge them into the previous file, delete the later files, and finally update the ETS record. As shown in the figure:

Queue structure:

The queue consists of rabbit_amqqueue_process and backing_queue. Rabbit_amqqueue_process is responsible for receiving messages, delivering messages to consumers, processing message acknowledgements, and so on. (the processing logic needs to call the interface) backing_queue is the concrete form and engine of message storage and provides the relevant interface to rabbit_amqqueue_process. If the destination queue for sending the message is empty and the queue has a consumer subscription, it is sent directly to the consumer, otherwise it needs to be sent to the queue temporary cache.

The queue has four states: a b g d 1. Alpha: index and message content are stored in memory, the most memory consumption is also the fastest, less consumption is CPU 2. Beta: index memory memory, message memory disk 3. Gama: index memory and disk, message storage disk 4. Delta: index and content are stored on disk, basically no memory consumption, consume more CPU and IUnio operation explanation: after the message is stored in the queue It is not fixed, it will continue to flow in the queue as the load of the system, and the state of the message will constantly change. Persistent messages, indexes, and contents must be saved on disk before they can reach any of the above states. In addition, in the third state, persistent messages will be available.

For ordinary queues, no priority and mirror are set, and the default implementation of backing_queue is rabbit_variable_queue, which reflects the status of messages through five sub-queues Q1, Q2, delta, Q3, and Q4. Q1 and Q4 contain only alpha status messages, Q2 and Q3 contain beta and gamma status messages, and Delta only contain delta status messages. In general, messages are sent to consumption and flow according to Q1-> Q2-> Delta- > Q3-> Q4, going through the process of memory to disk and disk to memory. As shown below:

However, not every message is the above process and will change according to the current load, as shown below: for message sending: if the load is low and memory is sufficient, the message arrives directly from Q1 to Q4 and will not pass through the disk (persistent messages will certainly go to disk, but also at Q1-> Q2-> Q3-> Q4) if the load increases and the memory is insufficient, the new message will go through Q1-> Q2-> Q3-> Q4. If the load is heavy and memory is completely insufficient, the index and message body of the new message are completely stored on disk, that is, queue delta.

For message consumption: 1, read from Q4, if there is a direct return. 2. Q4 is empty, while reading Q3, Q3, is empty. The return queue is empty. 3. Q3 is not empty. After taking out the message of Q3, determine the length of Q3 and delta. A, if both are empty, it is considered that Q2, delta, Q3, and Q4 are all empty, and the message of Q1 is moved to Q4. B, Q3 is empty delta is not empty, transfer the delta message to Q3. Transfer is read in segments according to the index file. Read the first paragraph first to determine whether the number of messages in this segment is equal to the length of delta, which means that all messages in delta can be transferred. In this case, the messages in Q2 and delta are transferred to Q3. Otherwise, only the messages read this time in delta will be transferred.

Follow the above steps to get the conclusion: when getting the message, if Q3 is empty, the whole team is listed as empty. Reason: the message eventually goes out from Q4, and Q3 will be fetched only if Q4 is empty, indicating that no message has been transferred to Q3 or Q4 since the last message was fetched. The above process also explains why message accumulation leads to significant performance degradation: because messages are written to deep queues, such as delta. Each write and read goes through the disk io.

The above is all the content of this article "what is the architecture in rabbitMq?" Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.