Shulou(Shulou.com)05/31 Report--

This article introduces how to compare RocketMQ with Kafka. The content is very detailed. Interested friends can use it for reference. I hope it will be helpful to you.

The internal trading system of Taobao uses the Notify message middleware independently developed by Taobao and uses Mysql as the message storage medium, which can be expanded horizontally. In order to further reduce the cost, we think that the storage part can be further optimized. In early 2011, Linkin opened up Kafka, an excellent message middleware. After Taobao middleware team did sufficient Review to Kafka, Kafka unlimited messages accumulated, efficient persistence speed attracted us. But at the same time, it is found that this message system is mainly located in log transmission, and there are still many features that are not satisfied with the use of Taobao transactions, orders, recharge and other scenarios, so we rewrite RocketMQ in Java language, which is located in non-log reliable message transmission (log scenario is also OK). At present, RocketMQ is widely used in order, transaction, recharge, flow calculation, message push, log streaming processing. Scenarios such as binglog distribution.

Data reliability

RocketMQ supports asynchronous real-time flushing, synchronous Replication, asynchronous Replication.

Kafka uses asynchronous flash disk mode, asynchronous Replication

Summary: the synchronous flushing disk of RocketMQ is more reliable than Kafka in terms of stand-alone reliability, and will not cause data loss because of the operating system Crash. At the same time, synchronous Replication is more reliable than Kafka asynchronous Replication, with no single point of data at all. In addition, the Replication of Kafka is in topic, which supports host downtime and automatic slave switching. However, there is a problem here. Because it is an asynchronous Replication, data will be lost after switching. At the same time, if Leader is restarted, it will cause data conflict with the existing Leader. The open source version of RocketMQ does not support Master downtime, Slave automatically switches to Master, and Aliyun version of RocketMQ supports automatic switching feature.

Performance comparison

Kafka writes TPS on a single machine in millions of messages per second, and message size is 10 bytes.

RocketMQ writes about 70, 000 messages per second to a single instance of TPS, deploys 3 Broker on a single machine, and can run up to 120000 entries per second, with a message size of 10 bytes.

Summary: the TPS of Kafka has reached millions on a single machine, mainly because the producer side merges multiple small messages and sends them to Broker in batches.

Why didn't RocketMQ do this?

Producer usually uses Java language and caches too many messages. GC is a serious problem.

Producer calls the API to send messages. The message was not sent to Broker and returned to the business successfully. When the Producer goes down, the message will be lost and the business will go wrong.

Producer is usually a distributed system, and each machine is sent by multi-thread. We believe that the amount of data generated by a single Producer in an online system is limited, not tens of thousands of data per second.

The function of cache can be completed by the upper layer business.

Number of queues supported by a single machine

When the Kafka has more than 64 queues / partitions, the Load will soar. The more queues, the higher the load, and the longer the response time for sending messages.

RocketMQ supports up to 50, 000 queues on a single machine, and Load will not change significantly.

What are the advantages of queues?

More Topic can be created on a single machine, because each Topic is made up of a batch of queues

The cluster size of Consumer is proportional to the number of queues. The more queues, the larger the Consumer cluster.

Real-time message delivery

Kafka uses short polling mode, and the real-time performance depends on the polling interval.

RocketMQ uses long polling, which is consistent with the real-time performance of Push, and the delivery delay of messages is usually a few milliseconds.

Consumption failed and retry

Kafka consumption failure does not support retry

RocketMQ consumption failure supports scheduled retry, with a smooth delay in the interval between each retry.

Summary: for example, if the top-up application calls the operator gateway at the current time and fails to recharge, it may be due to too much pressure on the other party, and the call will be successful later. For example, there is a similar demand for deduction from Alipay to the bank.

The retry here requires a reliable retry, that is, the failed retry message is not lost due to Consumer downtime.

Strict message order

Kafka supports message order, but when a broker goes down, messages will be out of order.

RocketMQ supports strict message order. In a sequential message scenario, when a broker goes down, sending messages will fail, but will not be out of order.

Mysql Binlog distribution requires strict message order

Timing message

Kafka does not support timing messages

RocketMQ supports two types of timing messages

The open source version of RocketMQ only supports timed Level

Aliyun ONS supports timing Level and specified delay time in milliseconds.

Distributed transaction message

Kafka does not support distributed transaction messages

Aliyun ONS supports distributed timing messages, and future open source versions of RocketMQ also have plans to support distributed transaction messages.

Message query

Kafka does not support message query

RocketMQ supports querying messages based on Message Id, and also supports querying messages based on message contents (specify a Message Key, any string, such as order Id, when sending messages)

Summary: message query is very helpful to locate the problem of message loss, for example, if an order failed to be processed, whether the message was not received or the processing error occurred.

Message backtracking

Kafka can theoretically retrace messages according to Offset.

RocketMQ supports retracing messages in time with a precision of milliseconds, for example, starting to re-consume messages at a certain time, minute and second before the day

Summary: typical business scenarios such as consumer do order analysis, but due to the failure of program logic or dependent system, all the messages consumed today are invalid and need to be consumed again from 0: 00 yesterday. Then the time-based message playback feature is very helpful to the business.

Consumption parallelism

The consumption parallelism of Kafka depends on the number of partitions configured by Topic. For example, if the number of partitions is 10, then a maximum of 10 machines can consume in parallel (each machine can only start one thread), or one machine can consume in parallel (10 threads consume in parallel). That is, the degree of consumption parallelism is the same as the number of zones.

The parallelism of RocketMQ consumption is divided into two situations.

The parallelism of sequential consumption is exactly the same as that of Kafka.

The degree of parallelism depends on the number of threads in Consumer. For example, if Topic is configured with 10 queues, consumed by 10 machines and 100 threads per machine, the parallelism is 1000.

Message locus

Kafka does not support message trace

Aliyun ONS supports message track

Develop language friendliness

Kafka is written in Scala

RocketMQ is written in Java language.

Message filtering on Broker side

Kafka does not support message filtering on the broker side.

RocketMQ supports two ways of message filtering on the broker side.

Filtering based on Message Tag is equivalent to the concept of sub-topic

Upload a piece of Java code to the server, you can filter messages in any form, and even do Message Body filtering and splitting.

Message stacking capability

Theoretically, Kafka has stronger stacking capability than RocketMQ, but RocketMQ stand-alone can also support 100 million message stacking capability. We believe that this stacking capability can fully meet business needs.

Open source community activity

Kafka community updates are slow

RocketMQ's github community has 250 individual and corporate users registered for contact information, and the QQ group has more than 1000 people.

Business support

Kafka's original development team set up a new company, but no related products have been seen so far.

RocketMQ has been open for public testing on Aliyun for nearly half a year. At present, it is available for commercial use free of charge in the form of cloud services, and promises users 99.99% reliability. At the same time, it completely solves the problem of operation and maintenance complexity of MQ products built by users themselves.

Maturity degree

Kafka is more mature in the field of logging.

RocketMQ has a large number of applications in Ali Group, producing a large amount of news every day, and successfully supporting many times of Tmall's 11 / 11 mass message test, which is a sharp weapon for data to cut peak and fill valley.

On how to compare RocketMQ and Kafka to share here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.