Shulou(Shulou.com)06/01 Report--

What this article shares to you is about which scenarios are suitable to use Apache TubeMQ, the editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

1. Foreword:

Apache TubeMQ is a new generation of MQ donated to Apache by Tencent after open source in 2019. It originates from Tencent's actual production environment and focuses on serving the high-performance storage and transmission of massive data. Today, MQ is already the Red Sea (Apache alone has 5 MQ). Compared with many open source MQ components, what are the characteristics of Apache TubeMQ, what scenarios do we apply, and what benefits can this product bring to us? Next, I would like to introduce from this point.

two。 In what scenarios is Apache TubeMQ applicable?

To sum up, the main scenarios are as follows:

The amount of data reported is too large for ordinary MQ to support.

Ordinary MQ can withstand, but it consumes too many machine resources, or the system is unstable and difficult to maintain.

The technology stack is pure JAVA, which is convenient for self-modification and version maintenance. even if there is something wrong with the version, you can quickly coordinate resources to stop losses immediately.

Functionally, as long as production and consumption, there is no need for transaction messages, Exactly Once and other advanced functions.

The system has a high degree of automation and tolerates a small amount of data loss in extreme cases.

3. Why is Apache TubeMQ suitable for these scenarios?

Personally, there are three main points:

3.1. From the needs of the actual production environment:

There are more than 40 trillion data-level services online over the past 8 years. Apache TubeMQ originated from the actual production environment of Tencent. At first, we also used Apache Kafka as most business samples for data services, but because the Kafka server was implemented by Scala, it was difficult to read and maintain, and the problem could not be solved immediately. At the same time, there are some unreasonable aspects in Kafka design, which are more complex to use. Under the pressure of the current network, based on the idea and actual needs of Kafka, we germinated a distributed message middleware with high reliability, high performance, low latency and low cost, which is based on server management and SAAS mode, and carried out product construction and continuous improvement around the positioning of this project.

So far, Apache TubeMQ has been focused on serving big data scenario for nearly 8 years, and now it has reached an average daily throughput of 40 trillion +, forming a relatively stable and easy to maintain time-tested products. The services include Tencent Ad solutions, PCG, Wechat and so on. Our largest cluster has more than 300 Broker, each Broker has 800 topic configurations, and the consumer group has a scale of nearly 3K.

Based on the needs of various businesses within Tencent, we are sure that this MQ is also suitable for everyone to use in similar business scenarios.

3.2. Strong enough high throughput and high performance indicators:

End-to-end latency below 10W TPS,10ms on a single machine. This is a performance stress test summary report made by Apache TubeMQ at the initial stage of open source. In general, under the TS60 model (10 Gigabit network card, 64G memory, 24-core CPU,12TSATA hard disk), configure 1000 topic, each topic is configured with 10 partitions, each message is 1K in size, under the premise of one production and two consumption. Stand-alone Broker can achieve more than 10W TPS (abbreviation of TransactionsPerSecond, number of successful request responses per second) and end-to-end latency below 10ms. This is only a conservative indicator given by us. I suggest you compare and test yourself in the same scenario. You can compare and test with any external MQ, and you will find something different.

Many MQ say that the system can achieve tens of millions of TPS, or even the end-to-end delay of 1ms. Do you think this indicator is too low? What I want to express is that the provision of indicators must have a supporting test premise. Under the given clear system configuration and production and consumption load, if you want to achieve 10 million-level TPS and more than 10W TPS on a single machine, the cluster only needs less than 100 Broker to scale out; without the above system configuration, production and consumption load, and reaching 10 million-level TPS, the machine level will be even lower.

3.3. Sufficiently transparent openness:

Apache TubeMQ is able to achieve the characteristics we have described, mainly due to its TubeMQ architecture built according to business scenarios. We not only use this system to support services internally, but also open source it to incubate projects in accordance with Apache rules, so that more external companies' business can use it to reduce costs and improve system performance and stability. The system is stable enough that students with MQ experience can run according to the guidelines on the official website; the system is completely open source and built with pure JAVA, even if the original team is no longer maintained, there are enough technical personnel in the market to support its improvement. According to the Apache community specification to operate the community, as long as you have any improvement suggestions, verify the effectiveness of the system can be integrated into the system, and the original team are domestic personnel, communication is more convenient.

4. What is the roadmap for the future development of Apache TubeMQ?

As an one-stop streaming data service platform, we want to open up the entire data reporting platform in this project, and organically integrate the modules of collection, aggregation, storage and forwarding involved in data reporting in the form of plug-ins (even TubeMQ can be replaced in this platform). Based on this system, the business only needs to publish and subscribe to data. It is easy to build analysis and application based on stream data. We are building various parts of the module, you are welcome to build together.

These are the scenarios that are suitable for using Apache TubeMQ, and the editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.