Shulou(Shulou.com)06/03 Report--

This article mainly explains "what are the solutions to the bottleneck encountered by MQ consumers besides horizontal expansion?" interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Next let the editor to take you to learn "MQ consumer side encountered bottlenecks in addition to horizontal expansion, there are any other solutions" bar!

1. Interview scenes and interview skills

During the recruitment season, a fan friend recently encountered such a question in Ant Financial Services Group's second round interview: what should I do if MQ consumption encounters a bottleneck?

Horizontal expansion, compared to many readers and my friend will blurt out, the interviewer will obviously not be satisfied with such an answer, and then asked: horizontal expansion is a pile of machines, is there any other way?

In the interview process, I suggest you think a little after hearing the question, do not give a direct answer immediately, but should discuss with the interviewer, on the one hand, you can have a deeper understanding of the interviewer's original intention of the question. at the same time, you can sort out your own ideas.

The consumer side has encountered a bottleneck, which is a result, but what is the reason for this result? It will look pale to talk about optimization and solutions without figuring out why.

In such an interview scene, how should we discuss communication? My thoughts are as follows:

Try to discuss with the interviewer how to judge the bottleneck on the consumer side.

How to find the root cause

Propose a solution

Warm Tip: for the rigor of this article, this article mainly takes RocketMQ as an example for analysis.

2. How to judge that the consumer side has encountered a bottleneck

In the field of RocketMQ consumption, there are usually two important indicators to judge that the consumer side is experiencing a bottleneck:

Number of message backlog (number of delays)

LastConsumeTime

In the open source console rocketmq-console interface, you can check the above two metrics on a consumer side, as shown in the following figure:

Delay: the number of messages overstocked, that is, how many messages are left unprocessed at the consumer end. The higher the value, the consumer side has encountered a bottleneck.

LastConsumeTime: indicates the storage time of the last successfully consumed message. The higher the value is from the current time, it also indicates that the consumer side has encountered a bottleneck.

Then why is there a backlog? Where is the bottleneck on the consumer side?

3. How to locate the problem

If there is a bottleneck at the consumer end, one of the easiest ways to identify whether it is a client problem or a server problem is to see whether other consumer groups in the cluster also have a backlog, especially whether there is a backlog in other consumer groups that subscribe to the same topic as the problematic consumer group. According to the author's experience, the backlog of messages is usually caused by the client, which can be proved by querying rocketmq_client.log:

Grep "flow" rocketmq_client.log

The emergence of logs such as so do flow control indicates that the consumption limit is triggered, which is directly triggered by the backlog of messages at the consumer end, that is, the consumer side cannot consume the pulled messages. To avoid memory leakage, RocketMQ will not continue to pull messages from the server and print the above log after the consumer side does not finish processing the messages.

So how to locate where the consumption is slow? Which line of code is stuck?

The usual troubleshooting method is to trace the thread stack, that is, to use the jstack command to see how the thread is running, so as to explore how the thread is running.

The commands commonly used are as follows:

Ps-ef | grep java jstack pid > j1.log

Usually for comparison, I usually print five files in a row, so that I can see whether the state of the same consumer thread has changed in five files, and if it has not changed, it means that the thread card owner is the main thread. that's what we need to focus on.

In RocketMQ, the consumer thread begins with ConsumeMessageThread_. By judging the thread, the following code is exciting:

The state of the consumer thread is RUNNABLE, but its state is the same in the five files, it can be concluded that the thread is stuck in the specific code, from the sample code is stuck in an external http excuse, so as to solve, usually involving external calls, especially http calls, you can set a timeout to avoid a long wait.

4. Solution

In fact, according to the third step, there is a high probability that you can identify where it is slow and encounter a performance bottleneck, usually nothing more than transferring third-party services, databases and other problems, and then prescribe the right remedy to the case. Database and other performance optimization is not within the scope of this article, so you can go so far here. Of course, the interviewer may continue to talk about database optimization and other topics in the future, so as to achieve communication and interaction with the interviewer. The technical exchange atmosphere is friendly, and the interview pass rate is greatly improved.

Finally, I would like to discuss with you a question: the backlog of information must mean that there is a consumption bottleneck. Does it have to be dealt with?

In fact, this is not the case. Let's recall why we need to use MQ, which is to use asynchronous decoupling and peak trimming to fill valleys. For example, during the Singles Day holiday, a large number of sudden traffic inflows are likely to lead to a backlog of messages. This is our intention to use MQ to resist sudden traffic, and the back-end applications consume slowly to ensure the stability of the consumer end. In the case of backlog, if the tps is normal, that is, it is not a big problem, the usual way to deal with this time is to expand the capacity horizontally, reduce the backlog as much as possible, and reduce the business delay.

At this point, I believe that everyone on the "MQ consumer side encountered bottlenecks in addition to horizontal expansion of what solutions," have a deeper understanding, might as well to the actual operation of it! Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.