Shulou(Shulou.com)06/03 Report--

This article mainly explains "the method course of maximizing the throughput of message queue". The content of the explanation in this article is simple and clear, and it is easy to learn and understand. let's study and study the method tutorial of message queue to achieve maximum throughput.

Some thoughts on Throughput

Write message queue throughput depends on the following two aspects

The best throughput is to fill up one of them, and in general, the private network bandwidth will be very high, so it is unlikely to be full, so naturally, the write speed of the message queue is full, so there are two points that need to be balanced.

Go-zero 's PeriodicalExecutor and ChunkExecutor are designed for this situation.

How many messages are written in bulk or how many bytes

How long is the write delayed?

Network bandwidth

Write speed of message queues (such as Kafka)

The throughput of consuming messages from a message queue depends on the following two aspects

A core problem here is that we cannot read too many messages into memory without considering the business processing speed, otherwise it may cause two problems:

Memory usage is too high, even OOM,pod has memory limit.

Messages accumulated when pod is stopped are not processed in time, resulting in message loss.

The reading speed of the message queue, in general, the reading speed of the message queue itself is fast enough compared to the speed of processing messages.

Processing speed, which depends on the business

Solution and implementation

To borrow a picture from Rob Pike, this is similar to queue consumption. Four gopher on the left are taken from the queue, and four gopher on the right are taken over for processing. The ideal result is that the rates on the left and right are basically the same, no one is wasted, no one is waiting, and there is no accumulation in the middle exchange.

Let's take a look at how go-zero is implemented:

Producer end

For {select {case

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.