Shulou(Shulou.com)06/02 Report--

I would like to share with you how to choose a message queue in high concurrency. I believe most people don't know much about it, so share this article for your reference. I hope you will learn a lot after reading this article. Let's learn about it!

In the scenario of high concurrency business, message queue plays an irreplaceable role in traffic peaking and decoupling. At present, the most commonly used message queues are RabbitMQ, RocketMQ, ActiveMQ, Kafka, ZeroMQ, Pulsar, etc.

There are so many message queues, which message queue should be chosen?

Select the basic criteria for message queuing

Although these message queues have their own advantages and disadvantages in terms of functions and features, we should have a basic standard when choosing them.

First of all, it must be an open source product. Open source means that if one day the message queue you use encounters a Bug that affects your system's business, at least there is a chance to fix or circumvent the Bug quickly by modifying the source code, instead of waiting for the developer to release the next version to solve the problem.

Secondly, this product must be popular in recent years and have a certain degree of community activity. The advantage of fashion is that as long as the usage scenario is not too unpopular, the probability of encountering Bug is very low, because most of the Bug encountered by others have already been encountered and fixed. Some problems encountered in the process of use, it is also relatively easy to search for similar problems on the Internet, and then quickly find solutions. Another advantage is that popular products will have a good integration and compatibility with the surrounding ecosystem.

Finally, as a passing message queue, several features are required:

Reliable delivery of messages: ensure that messages are not lost

Cluster: supports clustering to ensure that the service is not unavailable due to the downtime of a node, and that messages cannot be lost.

Performance: the performance is good enough to meet the performance requirements of most scenarios.

Next, take a look at the open source message queues that meet the above criteria to choose from.

RabbitMQ

First, let's take a look at the message queue RabbitMQ. RabbitMQ, published in 2007, is written in the Erlang programming language. It was originally designed for reliable communication between systems in the telecommunications industry, and is one of the few message queues that support the AMQP protocol.

RabbitMQ: lightweight, fast, its slogan, also clearly shows the characteristics of RabbitMQ: Messaging that just works, out-of-the-box message queue. In other words, RabbitMQ is a fairly lightweight message queue that is very easy to deploy and use.

One of the features of RabbitMQ is that it supports very flexible routing configuration. Unlike other message queues, it adds an Exchange module between Producer and Queue, which can be understood as a switch.

The function of the Exchange module is very similar to that of a switch, distributing messages sent by producers to different queues according to configured routing rules. Routing rules are also very flexible, and you can even implement routing rules on your own. If you just need this feature, RabbitMQ is a good choice.

The client of RabbitMQ supports probably the most programming languages of all message queues.

Next, let's talk about a few problems with RabbitMQ:

RabbitMQ's support for message stacking is not good, and when a large number of messages are overstocked, the performance of RabbitMQ will degrade sharply.

The performance of RabbitMQ is the worst of these message queues, processing tens of thousands to hundreds of thousands of messages per second. If the application has very high performance requirements for message queues, do not choose RabbitMQ.

RabbitMQ uses the programming language Erlang, and the cost of extension and secondary development is high.

RocketMQ

RocketMQ is Alibaba in 2012 open source message queue product, implemented in Java language, in the design reference to Kafka, and made some of its own improvements, and later donated to the Apache Software Foundation, officially graduated in 2017, became the top project of Apache. RocketMQ is widely used in Ali internal scenarios such as order, transaction, recharge, flow computing, message push, log streaming, Binglog distribution and so on. After many Singles Day tests, its performance, stability and reliability are reliable.

RocketMQ has good performance, stability and reliability, has almost all the functions and features that a modern message queue should have, and it is still growing.

RocketMQ has a very active Chinese community, and most of the questions can be answered in Chinese. RocketMQ is developed in Java language, and the source code is relatively easy to read, and it is easy to expand or redevelop RocketMQ.

RocketMQ has done a lot of optimization to the response delay of online business. In most cases, it can achieve millisecond response. If your application scenario is very concerned about response delay, you should choose to use RocketMQ.

The performance of RocketMQ is an order of magnitude higher than that of RabbitMQ, processing hundreds of thousands of messages per second.

The disadvantage of RocketMQ is that it is not integrated and compatible with the surrounding ecosystem.

Kafka

Apache Kafka is a distributed message publishing and subscribing system. It was originally implemented as a distributed log submission system by LinkedIn based on its unique design, and later became part of the Apache project.

In early versions, in order to achieve extreme performance, many design sacrifices were made, such as failure to guarantee the reliability of messages, possible loss of messages, lack of support for clustering, and poor functionality. These sacrifices are acceptable for the particular scenario of dealing with massive logs.

However, in the following years, Kafka gradually made up for these shortcomings, and the current Kafka has developed into a very mature message queue product, which can meet the needs of most scenarios in terms of data reliability, stability and functional features.

The compatibility of Kafka with the surrounding ecosystem is one of the best, especially in the field of big data and stream computing, almost all related open source software systems will give priority to supporting Kafka.

Kafka is efficient, scalable, and persistent. Its partition features, replicability and fault tolerance are all good features.

Kafka is developed in Scala and Java language, and batch and asynchronous ideas are widely used in the design, so that Kafka can achieve ultra-high performance. The performance of Kafka, especially the performance of asynchronous transceiver, is the best of the three, but there is no difference in magnitude with RocketMQ, which can process hundreds of thousands of messages per second.

When there are enough clients to send asynchronous batches concurrently and compression is enabled, the limit of Kafka's processing capacity can exceed 20 million messages per second.

But the problem with the asynchronous batch design of Kafka is that the response delay of synchronous sending and receiving messages is relatively high, because when the client sends a message, Kafka will not send it immediately, but will save a batch later and then send it. In its Broker, many places will use this design to save a wave first and then deal with it together. When there are not so many messages per second in your business scenario, the latency of Kafka will be higher. Therefore, Kafka is not very suitable for online business scenarios.

Message queue comparison

KafkaRocketMQRabbitMQ single machine throughput 100,000,000,000 level development language Java & ScalaJavaErlang message delay millisecond millisecond microsecond message loss parameter optimization configuration can achieve zero loss parameter optimization configuration can achieve zero loss has a low probability of loss consumption mode the impact of the number of PullPull+PushPull+Pushtopic on throughput topic reaches dozens, hundreds of times, the throughput will be greatly reduced topic to hundreds, thousands of hours There will be a small drop in throughput\ very high availability (distributed) very high (master-slave) high summary throughput, microsecond delay, distributed high availability, preferably supporting a small number of topic, there will be message duplication to support large-scale topic number, which is convenient for secondary development and expansion does not support cluster dynamic expansion, expansion and secondary development difficulties.

The above is all the contents of the article "how to choose message queues in High concurrency". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.