Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to analyze the Benchmark test of Apache TubeMQ. The content of the article is of high quality, so the editor shares it for you as a reference. I hope you will have some understanding of the relevant knowledge after reading this article.

1. Preface

Why is the Benchmark test item of Apache TubeMQ designed in this way? What feedback is given to each scene, and is it a targeted scene definition to magnify the advantages of TubeMQ? What are the similarities and differences between MQ and other Benchmark test cases? In this article, let's do the interpretation of TubeMQ's Benchmark test items and test results.

two。 Say in front.

TubeMQ's Benchmark test items and test results report tubemq_perf_test_vs_Kafka is included in the website of the project and is not directly displayed on the page. It is mainly discussed within the team at the time of open source and has no intention to compete, so you need to visit the page directly through the link.

There are 8 test scenarios only published in the tubemq_perf_test_vs_Kafka document, and each item contains multiple test items, with a total of nearly 100 items. The actual test report is more detailed than this, and the published test items and test results have given us a clear understanding of the actual situation of TubeMQ.

The test data of the latest version of TubeMQ must be better than the data done in the early days of open source, just because the initial test results have been published, there is no need to spend time and resources to adjust these contents, you can understand the concept of TubeMQ's Benchmark test items for more targeted analysis and review to verify its authenticity.

3. Design idea of TubeMQ Benchmark test item

When designing the Benchmark test item of Apache TubeMQ, we mainly consider the following points: what are the strengths and weaknesses of the core design scheme of the system, and what is its boundary value in application? How will we use this system in the real world, and what is the impact of the corresponding configuration adjustment on the system? As the load increases, how does the index trend of the system behave within the design allowable limits?

Each test scenario in the test report has been given in the way of [scenario, conclusion, data], and our metrics view is given. Here we still give it in the way of [scenario, data interpretation]. It is easy for everyone to view and refer to.

4. Interpretation of TubeMQ Benchmark test results: scenario 1: basic scenario, single topic, one-in-two-out model, using different consumption patterns and message packets of different sizes, the partition is gradually scaled out, and the performance of TubeMQ and Kafka is compared.

Through this test item, you can get a benchmark information: the Topic throughput of TubeMQ does not increase with the increase of the number of Partition, and the throughput has nothing to do with the number of Partition. At the same time, the throughput of a single storage instance of TubeMQ is not as good as that of Kafka single Partition. Combined with the design introduction of our storage and the test results of this data, we can see that Partition in TubeMQ is not an actual storage unit and needs to be distinguished from Kafka's Partiton concept when using it. In other words, the Partition of TubeMQ is a logical partition and is only linked to the parallelism of consumption.

Why use one production and two consumption preconditions: most test scenarios are one production and two consumption, why set this premise? From our point of view, the scenario of pure production is actually not in line with the actual online application scenario of MQ. There will be at least one consumption of online data, and at most dozens of copies. If only pure production is tested, it can not reflect the actual operation of the system after it is online. We have also done a pure production test, and we have also done a production of 1 consumption, a production of 2 consumption, and a production of more than one consumption. From the test data, the TPS (abbreviation of TransactionsPerSecond, successful request responses per second) of the system is directly affected by the number of consumption copies, while most of our environment reads 2 copies of data, so there are 2 consumption benchmark requirements in our main test scenario.

Why a single packet chooses 1K size as the front end: the selection of packet length is also obtained through this test case. Through the test data in scenario 1, we can see that as the packet length increases, although the traffic increases, the TPS of the system gradually decreases, and the larger the packet, the more TPS drops. From our test data, about 1 ~ 4K, the system is relatively acceptable in terms of TPS, cost and so on. Taking 1K as the benchmark, it is neither long nor short, which will be more in line with the actual use of the system.

Scenario 2: single topic case, one-in-two-out model, fixed consumption package size, number of scale-out instances, comparison of TubeMQ and Kafka performance

Here are a few pieces of information we can get from this table:

In scenario 1, although the throughput capacity of Apache TubeMQ's single storage instance is not as good as that of Kafka's single Parititon, when the number of storage instances is about 4, the same configuration of Kafka will be achieved.

According to the relationship between throughput capacity and the number of storage instances, the throughput of Kafka MQ increased from 1 to 10, while TubeMQ showed a downward trend.

Adjust the consumption mode (qryPriorityId, consumption priority ID) of consumers in TubeMQ, and the throughput capacity of the system will change differently. Online, you can adjust the consumption capacity of the consumption group according to the actual operation situation, provide differentiated services and improve the throughput capacity of the system per unit of resources.

Scenario 3: fixed message packet size, instance and partition number in multi-topic scenarios, and investigated the performance of TubeMQ and Kafka in 100,200,500 and 1000 topic scenarios

Why do you test so many Topic: some students have expressed this doubt, why don't we use a single Topic with more Partiton, or dozens of Topic tests, like other MQ Benchmark test items?

Based on the needs of online practical applications: there are hundreds or hundreds of Topic configured on the current network of TubeMQ, but our design capacity is 1000. We want to obtain the load curve of the system with the increase in the number of Topic through this scenario, so a few or dozens of Topic can not meet our actual needs.

In fact, the Benchmark test items done by various MQ are not quite in line with everyone's actual system applications, especially in big data's scenario. If we have dozens of machines in a cluster, will each Broker be configured with only a few Topic and Partition? Resource utilization cannot be improved if you can only configure dozens of Topic machines, so our benchmark is hundreds of Topic load tests.

We analyze and compare the stability of the system, the change of throughput, and the maximum possible situation of the system within the design range through the load of different scales. From the appendix of the document, we also give the traffic changes in different Topic scenarios. Through these, we can clearly know how the system behaves in practical application:

Scenario 4: 100 topic, one in one, five parts filtered out: one full Topic Pull consumption; filtering consumption uses five different consumption groups to filter out 10% message content from the same 20 Topic

The feedback in this scenario is the impact of Apache TubeMQ filtering consumption on the system. When online businesses construct a Topic, not every business will construct a Topic. Many of them are mixed in the same Topic for data reporting and consumption, so there will be a need to filter consumption data.

This use case gives a good feedback on the difference between client-side filtering and server-side filtering; at the same time, this indicator also shows that when filtering consumption, although the number of consumption has changed from 2 to 5, the throughput of the system has increased by about 50, 000 TPS, indicating that the load pressure on the system caused by filtering consumption is lower than that of full consumption.

4.5 comparison of TubeMQ and Kafka data consumption delay

Is Kafka end-to-end latency really like this? It seems to be different from what you use, some students have feedback in this post, "how to evaluate Tencent's open source messaging middleware TubeMQ?" ".

Why is there a difference? Because in order to improve the throughput of Kafka, we have adjusted the production configuration linger.ms of Kafka to 200ms TubeMQ batch.size to 50000 bytes. If we remove these two settings, the end-to-end delay of Kafka is similar to that of TubeMQ, but there is a big gap between its request TPS and the test results given in the test report. We have issued this test report before, in order to avoid unnecessary misunderstandings. We still keep this report data, because from the test parameter configuration we provided, the data is indeed like this:

If you are interested in this question, you can directly verify our test results and analysis under your own environment to see if this is the case.

Scenario 6: the effect of adjusting the memory cache size (memCacheMsgSizeInMB) of Topic configuration on throughput

What is reported in this scenario is the change that Apache TubeMQ brings to the system when the memory cache is resized. From the data, the size of the memory block will affect the throughput of TubeMQ. This is consistent with our design, the specific amount of impact and how much, we will introduce in a separate document.

4.7 scenario 7: the performance of the two systems in the case of serious consumption lag

Disadvantages of disk series: from this test, we can see that disk-based MQ has this disadvantage. The advantage of disk is that the cost is low, the number of reads and writes is better than SSD, and the disk may take a long time before the hardware is upgraded. This test also validates an idea of our original version to cache lagging data through SSD in exchange for the IO pull-up problem caused by disk lag.

The idea of caching lagging data through SSD is "how to evaluate Tencent's open source messaging middleware TubeMQ? "there have also been exchanges, and later we felt that this operation was not as fast as directly expanding the Broker node, so we abandoned the rollover SSD operation in the new version.

Through this test, we need to know clearly the performance of the system when it is read after the disk system occurs, and how to deal with it.

4.8 scenario 8: evaluate the performance of the two systems in the case of multiple models

Adaptability of different models: from this test, we can see that the throughput of TubeMQ increases greatly with the increase of memory and CPU in disk system, while Kafka will be much better when it comes to SSD model. Our analysis should be related to our read mode. When storage is not a bottleneck, Kafka's block read and write is much better than TubeMQ's block write random read. From this test, it is also clear that each system has its adaptability, and the key lies in the system's application environment and scenario.

This is the end of the Benchmark test on how to analyze Apache TubeMQ. I hope the above content can be helpful to you and learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.