Shulou(Shulou.com)05/31 Report--

This article introduces how to compare the message sending performance of Kafka, RabbitMQ and RocketMQ message middleware. The content is very detailed. Interested friends can use it for reference. I hope it will be helpful to you.

Introduction in distributed systems, we widely use message middleware for data exchange between systems, which is convenient for asynchronous decoupling. Now there are many open source message middleware, some time ago our own product RocketMQ (MetaQ kernel) is also smooth open source, get everyone's attention. So, which one has the best performance of message middleware? With this question in mind, our middleware test team compares three common types of messaging products (Kafka,.

Introduction

In distributed systems, we widely use message middleware to exchange data between systems, which is convenient for asynchronous decoupling. Now there are many open source message middleware, some time ago our own product RocketMQ (MetaQ kernel) is also smooth open source, get everyone's attention.

So, which one has the best performance of message middleware?

With this question in mind, our middleware test team compared the performance of three common messaging products (Kafka, RabbitMQ, RocketMQ).

Kafka is a LinkedIn open source distributed publish-subscribe messaging system, which currently belongs to the Apache rating project. The main feature of Kafka is based on the Pull model to deal with message consumption, the pursuit of high throughput, the initial purpose is for log collection and transmission. Version 0.8 starts to support replication, does not support transactions, and has no strict requirements for message repetition, loss and errors, so it is suitable for the data collection business of Internet services that generate a large amount of data.

RabbitMQ is an open source message queuing system developed in Erlang language and implemented based on AMQP protocol. The main features of AMQP are message-oriented, queue-oriented, routing (including peer-to-peer and publish / subscribe), reliability and security. AMQP protocol is more used in enterprise systems, requiring high data consistency, stability and reliability, followed by performance and throughput requirements.

RocketMQ is Ali open source message middleware, it is pure Java development, with high throughput, high availability, suitable for large-scale distributed system applications. The idea of RocketMQ originated from Kafka, but it is not a Copy of Kafka. It optimizes the reliable transmission and transactionality of messages. At present, it is widely used in trading, recharging, flow computing, message push, log streaming, binglog distribution and other scenarios.

Test purpose

Compare the performance of Kafka, RabbitMQ and RocketMQ in sending small messages (124bytes). This time, we only focus on the performance indicators of the server side, so the standard of the stress test is:

Continue to increase the pressure on the sender until the system throughput no longer increases and the response time increases. At this time, the performance bottleneck has appeared on the server side, and the corresponding optimal throughput of the system can be obtained.

Test scenario

In the synchronous sending scenario, the performance of the three message middleware is obviously different:

The throughput of Kafka is as high as 17.3w/s, which makes it the leader of high-throughput message middleware industry. This mainly depends on its queue mode which ensures that the process of writing to disk is linear IO. At this point the broker disk IO has reached a bottleneck.

RocketMQ also performed well, with throughput at 11.6w/s and disk IO% util approaching 100%. The message of RocketMQ is returned to ack after it is written into memory, and the operation of flushing disk is done by a separate thread, and all messages are written sequentially.

The throughput of RabbitMQ is 5.95wk, and the CPU resource consumption is high. It supports the AMQP protocol and implements a very heavyweight. In order to ensure the reliability of messages, a trade-off is made in terms of throughput. We also test the performance of RabbitMQ in the message persistence scenario, and the throughput is about 2.6w/s.

Test conclusion

In terms of server's performance of handling synchronous transmission, Kafka > RocketMQ > RabbitMQ.

Appendix: test Environment

The server is deployed on a standalone machine, and the machine configuration is as follows:

CPU24 core memory 94G hard disk Seagate Constellation ES (SATA 6Gb/s) 2000398934016 bytes [2.00TB] 7202 rpm network card 1000Mb/s

Application version:

Message middleware version Kafka0.8.2RabbitMQ3.5.4RocketMQ3.4.6 test script pressure side Jmeter java client message size 128byte concurrency can reach the maximum TPS on the server the optimal number of concurrent Topic partitions 8 flush strategy Kafka and RocketMQ are asynchronous, RabbitMQ Queue does not enable durable persistence, this is the comparison of how to send messages from Kafka, RabbitMQ and RocketMQ message middleware. I hope the above content can be of some help to you and learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.