Shulou(Shulou.com)06/03 Report--

This article mainly explains "what are the application scenarios and precautions of message queuing". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn "what are the message queuing application scenarios and precautions?"

What is message queuing

We can think of the message queue as a container for storing messages. when we need to use messages, we can just take messages out of the container for our own use.

Queue Queue is a first-in-first-out data structure, so messages are consumed sequentially.

Second, why use message queuing

Generally speaking, using message queuing can bring the following three benefits to our system:

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

Asynchronous processing to improve system performance and reduce response time through asynchronous processing

Traffic peaking to avoid high concurrent access to directly hang up the database

Application of decoupling to reduce system coupling

2.1. Asynchronous processing

Synchronous processing process

Asynchronous processing process

The result is returned immediately after the user's request data is stored in the message queue. Then, the system consumes the message. Because the user request data is returned to the user immediately after it is written to the message queue, but the request data may fail in subsequent business verification, writing to the database, and so on. Therefore, after using the message queue for asynchronous processing, it is necessary to modify the business process appropriately to cooperate. For example, after the user submits the order, the order data is written into the message queue, which cannot immediately return the successful submission of the user's order. After the order consumer process of the message queue has really processed the order, or even after it comes out of the library, the user needs to be notified of the success of the order by email or SMS to avoid transaction disputes. It's like we use mobile phones to book train tickets and movie tickets.

2.2 flow peaking

First, the transaction messages generated by short-time and high concurrency are stored in the message queue, and then the back-end services slowly consume these messages according to their own capabilities, so as to avoid directly destroying the back-end services.

For example, in some second kill and promotion activities of e-commerce, the rational use of message queue can effectively resist the impact of the influx of orders on the system at the beginning of the promotion activities. As shown in the following figure:

2.3 Application decoupling

Using message queuing can also reduce system coupling. We know that if there is no direct call between modules, then the new module or modified module will have less impact on other modules, so the scalability of the system is undoubtedly better.

Suppose there is such a scenario: system A sends data to three systems B, C and D, and sends it through interface calls. What if the E system also needs this data? What if system C doesn't need it now? the person in charge of system An almost crashed.

In this scenario, system An is seriously coupled with other messy systems, system A produces a more critical piece of data, and many systems need system A to send this data. System A should always consider the four systems B, C, D and E. what should I do if I fail? Do you want to resend it? do you want to save the message? My hair is white!

If you use the MQ,A system to generate a piece of data and send it to the MQ, which system needs to consume the data in the MQ. If the new system needs data, you can consume it directly from MQ; if a system no longer needs this data, you can cancel the consumption of MQ messages. In this way, system A does not need to think about who to send data to, maintain the code, or consider whether the call is successful, failure timeout, and so on. As shown in the following figure:

The producer (client) sends messages to the message queue, and the receiver (server) processes the messages. the system that needs to consume can directly go to the message queue to get messages for consumption without coupling with other systems. this obviously improves the expansibility of the system.

Message queuing works in a publish-subscribe mode in which message senders (producers) publish messages and one or more message recipients (consumers) subscribe to messages. From the figure above, we can see that there is no direct coupling between the message sender (producer) and the message receiver (consumer). The message sender sends the message to the distributed message queue to end the processing of the message. the message receiver obtains the message from the distributed message queue for subsequent processing and does not need to know where the message comes from. For the new business, as long as you are interested in this kind of message, you can subscribe to the message, which does not have any impact on the original system and business, so as to achieve the scalability design of the system business.

Third, some problems caused by the use of message queues

Reduced system availability: system availability is reduced to some extent, and the more external dependencies the system introduces, the easier it is to fail. Before joining MQ, we do not have to consider the loss of messages or the hanging of MQ, but after the introduction of MQ, we need to consider how to ensure the high availability of message queues, otherwise MQ hanging may cause the whole system to crash!

Increased system complexity: after joining MQ, we need to ensure that messages are not repeatedly consumed, deal with message loss, ensure the order of message delivery, and so on!

Data consistency problem: it is mentioned above that message queues can be asynchronous, and the async brought by message queues can really improve the response speed of the system. But what if the real consumer of the message does not consume the message correctly? This will lead to data inconsistencies!

IV. Comparison of commonly used message queues

There are many MQ products on the market, and the mainstream ones are Kafka, ActiveMQ, RabbitMQ and RocketMQ, but which one should we use when we do technology selection? Every MQ is not absolutely good or bad, that is, it depends on which scenario can be used to enhance its strengths and avoid weaknesses, make use of its strengths, and avoid its disadvantages.

Summary:

ActiveMQ: the community is relatively mature, but compared to the current, the performance of ActiveMQ is relatively poor, and the version iteration is very slow, so it is not recommended.

RabbitMQ: although slightly inferior to Kafka and RocketMQ in terms of throughput, because it is based on erlang development, it has strong concurrency, excellent performance, and low latency, reaching the level of microseconds. But also because RabbitMQ is based on erlang development, few domestic companies have the strength to do erlang source-level research and customization. If the concurrency requirement of the business scenario is not too high (100,000 or million), then RabbitMQ must be your first choice among the four message queues. If it is real-time computing, log collection and other scenarios in big data's field, using Kafka is the industry standard, absolutely no problem. The community is very active and will never be yellow, not to mention it is almost a factual norm in this field all over the world.

RocketMQ: produced by Ali, Java is an open source project, we can read the source code directly, and then we can customize our own company's MQ, and RocketMQ has the actual combat test of Alibaba's actual business scenario. The activity of the RocketMQ community is relatively general. At present, RocketMQ has been donated to Apache, but the activity on GitHub is not high, and the documentation is relatively simple. Then the interface is not in accordance with the standard JMS specification. Some systems need to be migrated and need to modify a lot of code. There is Ali's technology, you have to do a good job in case this technology is abandoned, the risk of community pornography, absolute confidence in their own company's technical strength, recommend using RocketMQ, otherwise go back to honestly use RabbitMQ, people have an active open source community, absolutely not yellow.

Kafka: the feature is obvious: it provides only a few core functions, but provides ultra-high throughput, ms-level latency, high availability, and reliability, and the distribution can be extended at will. At the same time, kafka is best to support a small number of topic to ensure its ultra-high throughput. The only disadvantage of kafka is that messages may be consumed repeatedly, which will have a slight impact on data accuracy. In big data's field and log collection, this slight impact can be ignored. This feature is naturally suitable for big data's real-time computing and log collection.

At this point, I believe you have a deeper understanding of "message queuing application scenarios and matters needing attention". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.