Shulou(Shulou.com)06/01 Report--

In this issue, the editor will bring you about how to judge the accumulation of messages in RocketMQ. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

One machine deployment

1. Machine composition

7 machines, all with 16 GB of memory

Each server has 4 CPU,2 cores

2. Running environment configuration

3. The way of brushing disk.

Each machine master machine adopts asynchronous brushing mode.

Second performance evaluation

1. Purpose of evaluation

Test whether there is a message stacking scenario in rocketmq.

2. Evaluation index

The difference between the maxOffset of producer sending messages and the currOffset of consumer consuming messages

The given constant message heap value.

3. Evaluation logic

If the difference value of the message offset is greater than the constant stacking value, it is considered that there is a message accumulation.

On the contrary, there is no message accumulation.

4. Evaluation process

(1) the producer sends a large number of messages to the queue with the name "orderTopicTest" of topic, which is set at 40000.

(2) consumer subscribes to topic with a specific name and consumes it. Each time a message is consumed, the Offset; of the current message is recorded and the attribute value of the maximum offset of the message is obtained from the current message object according to the "MAX_OFFSET" keyword, and then the difference between the offset MAX_OFFSET and offset is calculated. The key code is as follows:

(3) send messages and record the messages sent and their related logs.

If the difference of the message offset offset is greater than the given message accumulation value, a log is recorded, indicating that there is a message accumulation. On the contrary, there is no message accumulation. The resulting log is as follows

(4) message stacking processing

It can be seen from the log that 741 messages have been squeezed because the producer has been running for a long time.

As the consumer consumption service is enabled, messages are consumed at the same time, and the overall rate of message consumption is higher than the rate of message generation, so the difference value of message offset is constantly decreasing, so the second screenshot exists: the difference value of message offset is less than the threshold value 100, so there is a mixture of normal consumption and message accumulation.

Consumer continues to consume messages, and the rate of messages generated by producer can not keep up with the consumption rate of consumer, so the third figure is already normal message consumption, that is, the part of messages accumulated at this time has been consumed.

(5) points for attention

According to the documents on the rocketmq official website, in the case of a cluster with Slave, once Master discovers that Consumer accesses data accumulated on disk and accesses data accumulated on disk, it sends an instruction back to consumer, ordering consumer to pull data from slave, so that neither the consumer that sends messages nor the consumer that consumes messages will be affected.

The premise of this case is:

A) salve machines exist in the cluster

B) there is message accumulation in consumer

C) consumer accesses disk data for some reason (instead of accessing memory data such as pageCache)

The scenario of this situation is demanding and can only occur in scenarios with high concurrency. In addition, the cluster configuration of the production environment and message accumulation may also be due to disk size, network factors and other reasons. This test does not go deep into this scenario and is left for further testing.

Second, evaluation results

1. Message accumulation is a relative value. For consumer consumption messages, the difference between the largest maxOffset in a topic queue and the currOffset of the current consumption message is greater than a specific threshold before message accumulation occurs.

2. When the rate of sending messages is higher than the rate of message consumption, message accumulation may occur.

3. Other conditions are kept at a normal level, and the part of messages with message accumulation will continue to decrease over time until the messages are consumed.

4. In view of excessive accumulation of messages, you can choose to discard unimportant messages, that is, only log records without real consumption, so as to ensure the integrity of messages, so as to specially deal with the accumulation of messages.

The above is how to judge the accumulation of messages in the RocketMQ shared by the editor. If you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.