Shulou(Shulou.com)05/31 Report--

Editor to share with you how to achieve Rocketmq cluster consumption test, I believe that most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to know it!

One machine deployment

1. Machine composition

7 machines, all with 16 GB of memory

Each server has 4 CPU,2 cores

2. Running environment configuration

3. The way of brushing disk.

Each machine master machine adopts asynchronous brushing mode.

Second performance evaluation

1. Purpose of evaluation

Test the cluster mode consumption on the client side.

2. Evaluation index

(1) value of readQueueNums read queue associated with topic

(2) the number of consumer belonging to the same consumerGroup

(3) the total number of consumer consumption messages

(4) each consumer consumes messages and reads queue Id

(5) the number of master machines in the deployment cluster

3. Evaluation logic

If there are 5 queues and 2 consumer, then the first Consumer consumes 3 queues and the second consumer consumes 2 queues.

If the Consumer exceeds the number of queues, then the extra Consumer will not be able to consume messages.

The number of queues, the number of Consumer, and the Replance results are as follows

Number of queues Consumer number of Reblance result 52C1:3

C2:263C1:3

C2:31020C1-C10:1

C11-C20:0206C1:4

C2:4

C3-C4:3

4. Evaluation process

(1) check the number of queues associated with the topic on the server before sending the message.

(2) the producer side sends 20 messages to the topic queue with the name "clusterTopicTest". After sending the message, it records the basic information such as msgId, queueId, offset and so on.

(3) configure the consumer side to log the instanceName of each consumer side, the offset of the message, the queue queueId consumed, the body of the message, the message msgId, and the total number of messages consumed by each consumer.

(4) after each consumption, count the total number of all consumer consumption messages to determine whether the message has been lost.

(5) after each consumption, analyze the queueId of each consumer consumption queue to determine whether the queue has reached the load balance.

(6) Mark the number of queues of topic as An and the number of consumer as B, and make the following adjustments:

The first group: keep A unchanged, add B so that A > B, and then repeat steps 1-5.

The second group: keep A unchanged, add B so that A = B, and then repeat steps 1-5.

The third group: keep A unchanged, add B so that A = 2 * B, and then repeat steps 1-5.

The third group: add A, keep B unchanged, make 2 * A = B, and then repeat steps 1-5.

Group 5: reduce A, keep B unchanged, make 2 * A < B, and then repeat steps 1-5.

(7) Note: you need to start all servers first and send messages on the startup producer side, so that you can see the consumption of messages on each consumer side at the same time, because the rate at which messages are consumed is very fast.

(8) Note: the number of master machines and the number of queues of the specified topic on each master machine. Multiplying the two values is the final number of queues for rocketmq load balancing. (the number of master machines in step 6 is 2)

In the first group, a total of 20 messages were sent.

Number of queues Consumer number Reblance result

(expected) Reblance result

(actual) number of Master machine consumption

Master1Master2

85C1:4C1:4408C2:3C2:3123

C3:3C3:3033

C4:3C4:3303

C5:3C5:3033

Total number of 3 consumer consumption messages: 8 "3" 3 "3" 3 = 20

2 master machines, each topic has 8 queues, the expected number of queues is 2 "8" 16, and the actual number of queues is 4 "3" 3 "3" 3 = 16. It can be seen that the expected and actual queue distribution is the same.

Sending record of producer:

Consumer1's consumption record:

Consumer2's consumption record:

Consumer3's consumption record:

Consumer4's consumption record:

Consumer5's consumption record:

The second group, the total number of messages sent is 20

Number of queues Consumer number Reblance result

(expected) Reblance result

(actual) number of Master machine consumption

Master1Master2

88C1:2C1:2204C2:2C2:2022

C3:2C3:2022

C4:2C4:2022

C5:2C5:2022

C6:2C6:2204

C7:2C7:2202

C8:2C8:2202

Total number of 8 consumer consumption messages: 8 "3" 3 "3" 3 = 20

2 master machines, each topic has 8 queues, the expected number of queues is 2 "8" 16, and the actual number of queues 2 "2" 2 "2" 2 = 16. It can be seen that the expected and actual queue distribution is the same.

Total number of 8 consumer consumption messages: 4'2'2'2'2'2'2'4'2'2 = 20

Sending record of producer:

Consumer1's consumption record:

Consumer2's consumption record:

Consumer3's consumption record:

Consumer4's consumption record:

Consumer5's consumption record:

Consumer6's consumption record:

Consumer7's consumption record:

Consumer8's consumption record:

The third group, the total number of messages sent is 20

Number of queues Consumer number Reblance result

(expected) Reblance result

(actual) number of Master machine consumption

Master1Master2

84C1:4C1:4408C2:4C2:4404

C3:4C3:4044

C4:4C4:4044

Total number of 8 consumer consumption messages: 8 "3" 3 "3" 3 = 20

2 master machines, each topic has 8 queues, the expected number of queues is 2-8-16, and the actual number of queues is 4-4-4-4 = 16. It can be seen that the expected and actual queue distributions are the same.

Total number of 8 consumer consumption messages: 8, 4, 4, 4 = 20

Sending record of producer:

Consumer1's consumption record:

Consumer2's consumption record:

Consumer3's consumption record:

Consumer4's consumption record:

The fourth group, a total of 20 messages were sent.

Number of queues Consumer number Reblance result

(expected) Reblance result

(actual) number of Master machine consumption

Master1Master2

48C1:1C1:1103C2:1C2:1103

C3:1C3:1012

C4:1C4:1012

C5:1C5:1012

C6:1C6:1012

C7:1C7:1103

C8:1C8:1103

Total number of 8 consumer consumption messages: 8 "3" 3 "3" 3 = 20

2 master machines, each topic has 8 queues, the expected number of queues is 2'4'8, and the actual number of queues 1'1'1'1'1 = 8. You can see that the expected and actual queue distribution is the same.

Total number of 8 consumer consumption messages: 3 "3" 2 "2" 2 "2" 2 "3" 3 = 20

Sending record of producer:

Consumer1's consumption record:

Consumer2's consumption record:

Consumer3's consumption record:

Consumer4's consumption record:

Consumer5's consumption record:

Consumer6's consumption record:

Consumer7's consumption record:

Consumer8's consumption record:

Group 5, the total number of messages sent is 20.

Number of queues Consumer number Reblance result

(expected) Reblance result

(actual) number of Master machine consumption

Master1Master2

37C1:1C1:1013C2:1C2:1104

C3:1C3:1013

C4:1C4:1103

C5:1C5:1104

C6:1C6:1013

C7:0C7:0000

Total number of 8 consumer consumption messages: 8 "3" 3 "3" 3 = 20

2 master machines, each topic has 8 queues, the expected number of queues is 2-3-6, and the actual number of queues is 1-1-1-1-1-1-0 = 6. It can be seen that the expected and actual queue distribution is the same.

Total number of 8 consumer consumption messages: 3 "4" 3 "3" 4 "3" 0 = 20

Sending record of producer:

Consumer1's consumption record:

Consumer2's consumption record:

Consumer3's consumption record:

Consumer4's consumption record:

Consumer5's consumption record:

Consumer6's consumption record:

Consumer7's consumption record:

Second, evaluation results

1. In the rocketmq cluster consumption mode, subscription messages do achieve queue load balancer. The factors related to this load balancing consumption are the number of master machines and the number of queue of a specific topic. The total number of queues calculated by rocketmq is multiplied by these two values.

2. The cluster consumption capacity of rocketmq ensures the accuracy and completeness of messages. The total number of messages consumed is the same as the total number of messages sent by the producer, and there is no case of messages being discarded.

3. Analyze the consumer consumption log and show that each message will only be consumed once in different servers of the same consumerGroup group.

4. In the cluster consumption mode, if the total number of consumer exceeds the total number of queues, then the redundant server will not be able to consume messages.

These are all the contents of the article "how to achieve Rocketmq Cluster consumption Test". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.