Shulou(Shulou.com)06/01 Report--

This article mainly shows you "Why message queues are used in rocketmq". The content is simple and clear. I hope it can help you solve your doubts. Let me lead you to study and learn this article "Why message queues are used in rocketmq".

Why use message queuing

In fact, it is to ask you what are the usage scenarios of message queues, and then what are the specific scenarios in your project, and what are you using message queues in this scenario?

When the interviewer asks you this question, the expected answer is to say, what is the business scenario of your company and what are the technical challenges in this business scenario? it may be troublesome if you don't use MQ, but now you use MQ and bring you a lot of benefits.

Let's start with the common usage scenarios of message queuing. There are actually many scenarios, but there are three core scenarios: decoupling, asynchronism, and peaking.

Decoupling

Look at this scene. A system sends data to three BCD systems and sends data through interface calls. What if the E system also needs this data? What if the C system is no longer needed? The person in charge of system An almost collapsed.

In this scenario, system An is seriously coupled with other messy systems, system A produces a more critical piece of data, and many systems need system A to send this data. A system should always consider the four BCDE systems. What should I do if I fail? Do you want to resend it? do you want to save the message? My hair is white!

If you use the MQ,A system to generate a piece of data and send it to the MQ, which system needs to consume the data in the MQ. If the new system needs data, you can consume it directly from MQ; if a system no longer needs this data, you can cancel the consumption of MQ messages. In this way, system A does not need to think about who to send data to, maintain the code, or consider whether the call is successful, failure timeout, and so on.

Summary: through a model of MQ,Pub/Sub publish and subscribe messages, system An is completely decoupled from other systems.

Interview skills: you need to consider whether there is a similar scenario in the system you are responsible for, that is, a system or a module that invokes multiple systems or modules, and the calls between them are complex and troublesome to maintain. But in fact, this call does not need to call the interface synchronously, if it is possible to decouple it asynchronously with MQ, you need to consider whether you can use this MQ to decouple the system in your project. Show this in your resume and decouple it with MQ.

Async

Let's take a look at another scenario. System A receives a request and needs to write the library locally, and it also needs to write the library in three BCD systems. The local writing library needs 3ms, and the three systems need 300ms, 450ms and 200ms respectively. The total delay of the final request is 3 + 300450200 = 953ms, which is close to 1s, and the user feels that he is working on something. It is almost unacceptable for a user to initiate a request through a browser and wait for a second.

General Internet enterprises, for the direct operation of users, the general requirement is that each request must be completed within 200 ms, almost imperceptible to users.

If you use MQ, then system A sends 3 messages to the MQ queue in succession. If it takes 5ms for system A to accept a request and return the response to the user, the total time is 3 + 5 = 8ms. For the user, it actually feels like a click of a button, and 8ms will return directly later. Cool! The website is so good, so fast!

Peak clipping

From 0:00 to 12 every day, the system is calm, with 50 concurrent requests per second. As a result, the number of concurrent requests per second suddenly soars to 5k + from one to 12:00 to 13:00. But the system is directly based on MySQL, and a large number of requests pour into MySQL, executing about 5k SQL to MySQL per second.

For a general MySQL, it is enough to carry up to 2k requests per second. If the request reaches 5k per second, the MySQL may be directly killed, causing the system to crash and users will no longer be able to use the system.

But as soon as the peak is over, in the afternoon, it will be a trough, and it may be just one week of users operating on the site at the same time, and the number of requests per second may be only 50 requests, putting almost no pressure on the whole system.

If you use MQ, 5k requests per second are written to the MQ,A system to process a maximum of 2k requests per second, because MySQL processes a maximum of 2k requests per second. System A slowly pulls requests from MQ, pulling 2k requests per second, and ok no more than the maximum number of requests it can handle per second, so that system A will never fail even during peak hours. On the other hand, MQ has 5k requests coming in and 2k requests going out per second, resulting in a backlog of hundreds of thousands or even millions of requests in MQ during the noon peak (1 hour).

This brief peak backlog is ok, because after the peak, there are 50 requests per second into MQ, but system A still processes at the rate of 2k requests per second. So, as soon as the peak is over, system A will quickly resolve the backlog of messages.

What are the advantages and disadvantages of message queues

The advantages mentioned above are that there are corresponding advantages in special situations, such as decoupling, asynchronism and peak clipping.

The disadvantages are as follows:

Reduced system availability

The more external dependencies the system introduces, the easier it is to fail. Originally, you are the A system to call the interface of the three systems of BCD, and the four systems of human ABCD are fine, no problem, you add a MQ to come in, in case the MQ fails, MQ hangs, and the whole system crashes, aren't you finished? You can click here to see how to ensure the high availability of message queues.

Increased complexity of the system

If you add a MQ to it, how can you make sure that the message is not duplicated? How to deal with the loss of messages? How to ensure the sequence of message delivery? Big head, a lot of problems, a lot of pain.

Consistency problem

When system An is finished and returned successfully, people think that your request will be successful. But the problem is, if two systems of BCD succeed in writing the database of two BD systems, as a result, system C fails to write the library, what to do? Your figures are inconsistent.

So message queuing is actually a very complex architecture, and you have a lot of advantages to introduce it, but you also have to do all kinds of additional technical solutions and architectures to avoid it. After you do it, you will find that, my God, the complexity of the system has increased by an order of magnitude, maybe 10 times more complex. But at the critical moment, you still have to use it.

What are the advantages and disadvantages of Kafka, ActiveMQ, RabbitMQ, and RocketMQ?

Characteristic ActiveMQRabbitMQRocketMQKafka single machine throughput 10,000, an order of magnitude lower than RocketMQ, Kafka as ActiveMQ10 10,000, support high throughput 100000, high throughput, generally cooperate with big data system to carry out real-time data calculation, log collection and other scenarios the impact of topic quantity on throughput

Topic can reach the level of hundreds / thousands, and the throughput will decline slightly, which is one of the major advantages of RocketMQ. Under the same machine, it can support a large number of topictopic from tens to hundreds of times, and the throughput will drop significantly. Under the same machine, Kafka tries to ensure that the number of topic is not too large. If you want to support large-scale topic, you need to add more machine resources, timeliness, ms, microseconds, which is a major feature of RabbitMQ. The lowest latency ms level delay is high within the ms level, high availability based on master-slave architecture is very high as ActiveMQ, distributed architecture is very high, distributed architecture is very high, multiple copies of one data, a few machines are down, and data will not be lost, which will not lead to unavailable messages with low reliability and low probability of data loss basically without parameter optimization. Can achieve zero loss and RocketMQ function support MQ domain function is extremely complete based on erlang development, strong concurrency, excellent performance, low latency MQ function is more perfect, or distributed, scalable function is relatively simple, mainly support simple MQ function, real-time computing and log collection in big data field are used on a large scale

To sum up, after various comparisons, there are the following suggestions:

General business systems want to introduce MQ, at first everyone uses ActiveMQ, but now it is true that people do not use much, without the verification of large-scale throughput scenarios, and the community is not very active, so forget it. I personally do not recommend this.

Later, people began to use RabbitMQ, but it is true that the erlang language prevents a large number of Java engineers from studying and controlling it. For the company, it is almost out of control, but it is true that others are open source, stable support and high activity.

But now it is true that more and more companies will use RocketMQ, which is really good (produced by Ali), but the community may have the risk of suddenly becoming yellow. Those who have absolute confidence in their company's technical strength are recommended to use RocketMQ, otherwise go back to honestly use RabbitMQ, others have an active open source community, absolutely will not be yellow.

So small and medium-sized companies, technical strength is relatively general, technical challenges are not particularly high, using RabbitMQ is a good choice; large companies, infrastructure research and development strength is strong, using RocketMQ is a good choice.

If it is real-time computing, log collection and other scenarios in big data's field, using Kafka is the industry standard, absolutely no problem. The community is very active and will never be yellow, not to mention it is almost a factual norm in this field all over the world.

The above is all the content of the article "Why message queuing is used in rocketmq". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.