Shulou(Shulou.com)06/01 Report--

Today, I will talk to you about what points you should pay attention to when using Apache TubeMQ, which may not be well understood by many people. in order to make you understand better, the editor has summarized the following for you. I hope you can get something according to this article.

1. Preface

What points should you pay attention to when using Apache TubeMQ? Why is it designed like this? I would like to summarize and answer here for you to understand.

two。 Q2.1 Why should Broker be configured before use?

This should be the biggest difference between Apache TubeMQ and other MQ in terms of user experience. As shown in the following figure, before a cluster can deploy Master;Master and deploy Broker online, it is necessary to configure Broker-related configuration in Master and adjust its status to online before Broker can join the external service of the cluster. Why is TubeMQ designed in this way? why can't it be like other MQ, such as Kafka, that after configuring the Broker file, the process can be started and used directly in the cluster?

For cluster control: you can imagine that if you join directly, there are any shortcomings, although it is convenient to use, but if I, as a counterfeiter, can unwittingly join the cluster to make the system run in chaos, resulting in data leakage; or some people may say that authentication and authorization is added to the startup configuration file, but if this node dies, where can I easily understand the overall situation of each node online? TubeMQ uses SAAS mode for cluster management, which requires a clear understanding of the current state of each Broker node in the cluster and a place to centrally manage them. In order to achieve this effect, we have designed a state machine:

The main body status of Broker is draft-> online-> offline, and the newly added Broker configurations are all draft. The new version of Broker process can be started normally only after the Broker status is set to online and later. After the Broker configuration changes, Broker loading operation is required to make the configuration take effect. When broker is offline, you only need to call the Broker offline operation. There are other areas that we need to pay special attention to when operating the state machine:

The flow of each state has a state machine. When operating, avoid operating on all Broker nodes in the cluster at the same time. Only when the running child state is idle can the next batch of nodes be processed.

After the Topic-related configuration changes, the modified Broker has to be overloaded before the configuration can take effect. The [number of partitions, publishable and subscripable] changes to the Topic configuration will trigger the flow state machine, and the changes to these parameters should be carried out in batches.

Through the above design, we realize the state management and configuration management of the whole node.

2.2 Why does Topic not support automatic addition and enter authorization codes when adding new ones?

Permission control: similar to Broker management, using SAAS mode, resources are centralized and limited, and the business needs to split the resources in advance and then use them when ready. At the same time, all the change operations on the page, and some queries exceeding the authorization code, are hierarchical from the point of view of system management:

Our internal operation and management system is operated by operation and maintenance. The authorization code is relatively simple. It is placed in the confModAuthToken parameter of the master.ini file, and whoever has the permission to log in to the host has the permission to operate. You can also adjust this area according to your actual situation, such as your authentication system.

When specifying the Topic configuration, our API is required to specify the BrokerId set to configure the Topic. Its function is to accurately allocate the Topic to the specified Broker to avoid load inconsistency. After adding the Topic configuration, we also need to Reload the Broker collection of the new Topic. Our approach is to Reload the Broker in batches, and then do the Reload of the next batch of Broker after the previous batch of Broker Reload, to complete the overall configuration changes.

2.3 Why do I have to Reload every configuration change for such a long time?

This should also be one of the biggest differences in the user experience between Apache TubeMQ and other MQ. TubeMQ makes configuration changes through the Reload operation of Broker. A Broker initiates the change by calling Reload to the end of the change. The whole process takes about 1.2 minutes. Why do TubeMQ changes do this? why not write configuration data directly to zk, like other MQ, such as Kafka and Pulsar? So the configuration can take effect in real time?

The figure above is a complete state transition diagram of Reload. The Reload operation of Broker actually consists of the following nine steps:

Execute the Reload command, and Master sets the acceptPublish status of the specified Broker to false, pauses production first, and waits for a maximum heartbeat waiting period (25 seconds)

Receive a change notification to the Producer of the Broker production data, suspend sending data to the Broker, and wait for the configuration to be completed

Master continues to specify the acceptSubscribe of Broker as false, pauses consumption, and waits for a maximum heartbeat waiting period

The configuration of the Consumer-aware Broker that subscribes to the Broker data will change, suspend consumption, and wait for the configuration to complete

Broker completes the configuration changes within 8 seconds of the above process.

Master specifies the acceptSubscribe of Broker as true, starts the consumption of this Broker, and waits for a maximum heartbeat cycle.

Consumer initiates a subscription to the Broker

Master specifies that the acceptPublish status of the Broker is set to true, start the production of the Broker, and wait for a heartbeat cycle

Master sets the administrative state to idle.

Why would you do that? This operation takes a long time and complicated.

To avoid data loss: as you can see from the process, TubeMQ's Reload uses this similar system restart process to allow consumption to join the cluster as much as possible to allow consumption to join the cluster before production. When the system capacity is expanded, when Consumer is set to [start consumption from the maximum location for the first time by default, subsequent consumption], the data loss occurs when the partition information is obtained by Producer before Consumer and the production data is obtained.

Some students may say, why not write metadata to the initial offset location of ZooKeeper to solve this problem, just like Pulsar?

Decoupling from ZooKeeper: as you can see from the system implementation, ZooKeeper supports the persistence of offset in TubeMQ. After the system is actually running, it can exist without ZooKeeper. If we first write metadata into ZooKeeper to set the initial offset location when we expand capacity, the system will become dependent on ZooKeeper. For example, data will still be lost when Master is not connected to ZooKeeper. At the same time, this design will increase the processing logic of configuration changes. And the expansion time is prolonged, and the expansion operation may even fail due to ZooKeeper exception.

Considering that the online Topic operation does not need to be generated immediately, and with the development of the system, different types of configurations may need to do state machine management and multiple changes are operated together, there are still cases where multiple heartbeats can be synchronized before they can be updated. Therefore, at present, we have adopted the asynchronous mode mentioned above.

Note: from the above description, we can see that when Broker carries out Reload, there will be a brief stop reading and writing operation. Therefore, for online use, we are all batch-by-batch Broker operations that need to update the configuration, batch by batch Broker updates to complete Topic configuration updates.

2.4 Why can't the 0.8.0 client connect to the 0.3.0 server?

SAAS control: Apache TubeMQ uses strong control server mode. Online, we must use the server high-version client to allow low-version client to exist; that is, the server is compatible with the low-version client, but the high-version client is not compatible with the low-version server, so as to solve the problem of client complexity.

2.5 Why is the API operation of WEB not like that of Restful? Why do you want to manually splice the query return of WEB API?

Convenience and performance considerations: anti-Restful mode mainly considers that new modification operations can be completed by entering url directly into the browser. The disadvantage is that the number of modification operations may only be up to a few hundred records at a time (maximum URL length limit). The return result of manual stitching json is mainly based on performance considerations. From our tests, manual operation can achieve millisecond time consumption, and the use of components may be seconds. The problem is that special attention needs to be paid to coding.

2.6 where can I find indicator data, such as production and consumption in each district?

Under the Log log directory, production metrics are stored in get_transfer.log and consumption metrics are stored in put_transfer.log, refined to each partition granularity. As the load of our system is really high, the schemes adopted are all spit targets, relying on third-party components to complete the corresponding data deep processing.

2.7 what is the direction of the functions of the subsequent TubeMQ?

Personally, it depends on the trend of the open source community: Apache TubeMQ has now been donated to Apache, although we are the original team of the project, but we are only participating in the community. From our point of view, we will meet our internal business needs and contribute our characteristics to the community, so we welcome you to work together to promote the prosperity of the community.

After reading the above, do you have any further understanding of what points you should pay attention to when using Apache TubeMQ? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.