Shulou(Shulou.com)05/31 Report--

The purpose of this article is to share with you how to test the offset Offset equilibrium distribution of RocketMQ queue queue. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

One machine deployment

1. Machine composition

7 machines, all with 16 GB of memory

Each server has 4 CPU,2 cores

2. Running environment configuration

3. The way of brushing disk.

Each machine master machine adopts asynchronous brushing mode.

Second performance evaluation

1. Purpose of evaluation

Test queue accept message load balancing

2. Evaluation index

After receiving messages for each queue, the offset offset is roughly the same.

3. Evaluation logic

Create a topic, and configure the number of queue under the topic (3, 5, 8, and 16). After sending a message, print the corresponding offset of the message, increasing the amount of offset compared to the message.

4. Evaluation process

(1) create the name "topicQueueOffsetTest" on the master machine

(2) the topic configuration file created by the console is saved in the store directory. Check the / root/store/config/topic.json file to find the original data of the topic.

(3) start 5 threads on the client side to send different numbers of messages. After sending messages, record the offset of the messages in each queue queue.

Queue is configured with a default value of 8, and the number of messages sent is 550,250 messages.

Queue is configured with a default value of 8, and the number of messages sent is 5 "400" 2000 messages.

The client sets the number of queue queues to 12, sends the number of messages again, and records the offset of the messages. The key code is as follows

The queueNum=12 shown in the log here refers to the number of queue obtained by the producer on the client side, but subsequent logs show that the queueID on the server side is still 0-7, with a total of 8, and the number of the two types of queue is not equal.

When producer sends messages, modifying the value of the defaultTopicQueueNums attribute of the configuration file does not work for the previously running borker server, and the service needs to be restarted to make the number of queue of the running topic really effective.

There are two ways to dynamically change topic and topic-related properties

First, edit the / root/store/config/topic.json file of the master machine, find the data with the topic name "topicQueueOffsetCheck", update its readQueueNums and writeQueueNums properties, and restart the master cluster and the slave cluster.

The second method: dynamically update topic-related data in the rocketmq console (after this method is changed, the topic data will be automatically synchronized to other master and other slave, without having to restart the master and slave services). Here, I use the second way to update.

Update the queue of server to 12, send 2000 messages again, and find that the message offset of the old and new queues is basically balanced.

The queues with queueId 0-7 have more messages, the message offset of each queue is almost the same, and the message load is balanced.

Queues with a queueId of 8-11 have fewer messages and are four new queues, and the offset between the four queues is basically balanced.

Throughout the messages saved in these 12 queues, the last offset offset of the queueId=0 queue was 508, this offset=594, the difference is 594-508-86, and the message differences of the other quereId are basically about 83. It indicates that the queueNums is updated dynamically. After horizontal expansion, the queue queue can store messages evenly after receiving messages.

It is analyzed from this example that the balanced distribution of messages received by queue means that the increase of messages received by each queue can be balanced; it does not mean that the offset of the newly added queue queue after capacity expansion needs to be increased from 0 to the offset of the original queue, while the original queue has to wait until the message offset offset of all queue reaches the same level.

Keep the queueNums=12 unchanged, increase the number and number of threads, send 6000,000,18000 messages, record the message offset again, and the final result is as follows: the new offsets of all queue's "accept messages" can be balanced.

Keep the queueNums=12 unchanged, increase the number and number of threads, send 6000400024000 messages, and record the brokerName and queueId that save the messages.

By analyzing the logs, it can be concluded that messages are indeed evenly distributed among the queues of the broker-master1 and broker-master2 machines.

Second, evaluation results

1. The message sent by the client. After the server cluster receives the message, it can be evenly distributed to each master machine in the cluster, and the message received by each queue on each machine is also evenly distributed.

2. Dynamically increase the number of queueNums. After horizontal expansion, the number of messages received by the new and original queue can also achieve a balanced distribution.

3. When the server creates a topic, it sets the default queueNums value, which has a higher priority than the defaultQueueNums set by creating the producer.

4. For the already running topic, if you need to update the relevant attributes of topic dynamically, it is recommended to use the console of rocketmq to update dynamically.

Thank you for reading! On "how to achieve RocketMQ queue queue offset Offset equilibrium distribution test" this article is shared here, I hope the above content can be of some help to you, so that you can learn more knowledge, if you think the article is good, you can share it out for more people to see it!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.