Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "Why the autoCreateTopicEnable in RocketMQ can not be set to true". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

1. Phenomenon

Many netizens will ask why there are multiple Broker servers in the cluster and autoCreateTopicEnable is set to true, which means automatic creation of Topic is enabled, but the routing information of the newly created Topic is only included on one of the Broker servers. Why?

Expected value: for the high availability of message delivery, you want the newly created Topic to create a corresponding queue on each Broker in the cluster to avoid the single node failure of the Broker.

The screenshot of the phenomenon is as follows:

As shown in the figure above, the routing information for the automatically created topicTest5:

TopicTest5 created queues only on the broker-a server, but not on the broker-b server, which did not meet expectations.

The default number of read and write queues is 4.

Let's take a look at the RocketMQ default topic routing information screenshot as follows:

From the figure, you can default the routing information of Topic to 8 queues on broker-an and 8 queues on broker-b.

2. Thinking

How is the routing information for the default Topic created?

Where is the routing information for Topic stored? Nameserver? Broker?

What is the default number of queues for RocketMQ Topic?

3. Principle 3.1 RocketMQ basic routing rules

Broker registers the routing information stored on the server with Nameserver at startup, sends heartbeats to Nameserver every 30 seconds, and updates the routing information.

Nameserver scans the routing table every 10 seconds and removes the corresponding routing information if Broker service downtime is detected.

The message producer will re-pull the routing information of the Topic from the Nameserver and update the local routing table every 30 seconds; before the message is sent, if the routing message of the corresponding topic does not exist in the local routing table, it will actively pull the message of the topic to Nameserver.

Going back to the topic of this article: autoCreateTopicEnable, turn on automatic topic creation. Imagine how to deal with this situation if any of the above steps cannot get routing information for a non-existent topic when the producer sends a message to a non-existent topic.

In RocketMQ, if autoCreateTopicEnable is set to true, the message sender will try to use a system default topic name (MixAll.AUTO_CREATE_TOPIC_KEY_TOPIC) when the routing message for the NameServer query topic is empty, and the routing information obtained by the message sender is:

But here comes the problem: the default Topic creates eight queues on each Broker in the cluster, so the question is, why does the newly created Topic only create four queues on one Broker?

3.2Inquiry into autoCreateTopicEnable mechanism 3.2.1 timing of default Topic route creation

Warm Tip: this article will not track the whole creation process in detail, but will only point out the key entry points of the source code. if you want to learn more about the highly available implementation principles of NameServer routing messages and message sending, it is recommended to consult the second and third chapters of the author's book "RocketMQ Technology Insider".

Step1: in the Broker startup process, the TopicConfigManager object is built. The construction method first determines whether automatic theme creation is enabled. If automatic theme creation is enabled, the routing information of the default theme is added to the topicConfigTable. TopicConfigManager construction method

Note: all the routing information in this topicConfigTable will be sent to Nameserver with Broker in the heartbeat packet. After Nameserver receives these information, the routing information table of the corresponding Topic will be updated.

The defaultTopicQueueNum for BrokerConfig defaults to 8. Both Broker servers run the above process, so eventually the routing information about the default topic in the Nameserver will contain eight queues for each of the two Broker.

Step2: the producer looks for routing information. The producer first queries NameServer for routing information. Since it is a non-existent topic, the returned routing information is empty. RocketMQ will use the default topic to find it again. Since automatic creation of routing information is enabled, NameServer will return the routing information of the default theme to the producer. Then select a queue (default polling) from the returned routing information. Will the number of queues change after the message sender gets the queue information of the default Topic from Nameserver? The answer is yes, the code is as follows:

MQClientInstance#updateTopicRouteInfoFromNameServer

Warm reminder: when the message sender arrives at the default route information, the number of queues returned by DefaultMQProducer#defaultTopicQueueNums and Nameserver will be the minimum, and the default value of DefaultMQProducer# defaultTopicQueueNums is 4, so the number of queues for automatically created topics defaults to 4.

Step3: sending messa

DefaultMQProducerImpl#sendKernelImpl

In the request message when the message is sent, set the default topic name, the message sending topic name, and the number of queues used is DefaultMQProducer#defaultTopicQueueNums, that is, the default is 4.

The processing flow after receiving the message on the Step4:Broker side the server receives the message and sends the processor: SendMessageProcessor. When the message is sent, the super.msgCheck method: AbstractSendMessageProcessor#msgCheck is called.

On the broker side, TopicConfigManager is first used to query the routing information according to the topic. If the routing configuration (routing information) of the topic does not exist on the Broker side, then if the routing configuration information of the default topic exists in the Broker, the routing information of the new Topic is created in the Broker according to the number of queues in the message sending request. In this way, there will be routing information for the topic on the Broker server.

The routing information that exists in the topic configuration manager on the Broker side will send a heartbeat to the Nameserver and report to the Nameserver, on the other hand, there will be a scheduled task, regularly stored in the broker side, the specific path is ${ROCKET_HOME} / store/config/topics.json, so that when Broker is shut down and then restarted, the routing information will not be lost.

Readers, when you follow up to this step, you should give the creation of the topic sequence diagram for the convenience of how the routing information of the new topic is created when the automatic topic creation mechanism is enabled:

3.2.2 phenomenon analysis

After the above process of automatically creating routing mechanism, we can easily analyze and draw the following conclusion: because automatic creation of routing information is enabled, message senders cannot get routing information according to Topic going to NameServer, but then according to the default Topic, they can get routing information from NameServer (there are eight queues in each Broker), because both Broker will report routing information to NameServer at startup. In this case, the routing information cached by the message sender is 2 Broker, and each Broker defaults to 4 queues (for reasons, see 3.2.1:Step2 's analysis). According to the polling mechanism, the message sender then sends the first message selection (messageQueue:0 of broker-a) and sends a message to Broker. When the Broker server processes the message, it will first check the routing information in its own routing configuration manager (TopicConfigManager). At this time, there is no corresponding routing information, and then try to query whether there is a default Topic routing information. If so, autoCreateTopicEnable is enabled. Then the routing information of the new Topic is created in TopicConfigManager, which exists in the memory with the Broker server, and then the message is sent to an end. At this point, there is no routing information for the newly created Topic in NameServer.

Here are three key points:

When autoCreateTopicEnable is enabled to create a topic, the topic is created on the broker side and only when the message producer sends a message to the broker side.

Then the Broker side will send the newly created routing information to the NameServer within a heartbeat packet cycle, and at the same time, the broker side will have a scheduled task to persist the routing information in memory to the disk on the broker side.

The message sender will update the routing information to the NameServer every 30 seconds. If the message sender does not send the message within a period of time, there will be no message to send the second Broker in the cluster, then the routing information of the newly created Topic in the NameServer will only contain Broker-a, and then the message sender will pull the latest routing information to the NameServer. At this time, the message sender will cache the routing information of two broker, which will become the routing information of a Broker. The above phenomenon occurs when a message for that Topic is never sent to another Broker.

This is the end of the reason. Now we can boldly assume that if the autoCreateTopicEnable mechanism is enabled, queues will be created on both Broker. In fact, we only need to send 9 messages in a row and quickly to create queues on both Broker. The verification code is as follows:

Public static void main (String [] args) throws MQClientException, InterruptedException {DefaultMQProducer producer = new DefaultMQProducer ("please_rename_unique_group_name"); producer.setNamesrvAddr ("127.0.0.1 args 9876"); producer.start (); for (int I = 0; I < 9; iBytes +) {try {Message msg = new Message ("TopicTest10", "TagA", ("Hello RocketMQ" + I) .getBytes (RemotingHelper.DEFAULT_CHARSET)) SendResult sendResult = producer.send (msg); System.out.printf ("% s% n", sendResult);} catch (Exception e) {e.printStackTrace (); Thread.sleep (1000);}} producer.shutdown ();}

The verification result is shown in the figure:

This is the end of the content of "Why can't autoCreateTopicEnable in RocketMQ be set to true"? thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.