Shulou(Shulou.com)06/03 Report--

This article introduces the knowledge of "how to design a highly concurrent message middleware". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Knowledge points involved in message middleware

If you want to design a message middleware with high concurrency, you must first understand the specific knowledge points involved in the message middleware. In general, designing a good message middleware requires at least the following conditions:

Producer and consumer model.

Support for distributed architecture.

High availability of data.

Message data is not lost.

Next, let's talk about these technical points separately for message middleware.

Producer-consumer model

I believe that many partners have a better understanding of producer and consumer models, to put it simply: message middleware can not only enable other applications to produce messages, but also enable other applications to consume corresponding messages.

For producer and consumer models, we have more issues to consider. Next, I will guide you to think step by step.

First of all, let's consider the question: if the producer produces the message, how should the message middleware store the corresponding data? Stored in memory? Stored on disk? Or is it stored in memory and disk at the same time?

What do we do with message data if we store it in memory and disk at the same time? After the producer delivers the message to the message middleware, we write the data to disk immediately? Or should the data first reside in memory and then be brushed to disk at regular intervals? If it is brushed to disk at regular intervals, then we have to consider the problem of disk file segmentation, that is to say, how many disk files do we need to divide the message data into? You can't put all the data in one disk file. If you need to split into multiple disk files, what are the rules for slicing?

These are all the problems we need to consider when designing a message middleware. However, this is only a small part of the problem. If you want to stand out in the interview, you need to move on, and there are some important questions to pay attention to.

If the file is split into multiple disk files according to certain rules, does it also need to manage metadata to identify specific messages of the data (such as the DataNode metadata information stored in the NameNode node in Hadoop, through which the NameNode node can better manage the DataNode node)? The metadata can include: the offset of the message data, or it can be the unique ID of the message data.

After considering the problem of data storage, we also need to consider: how does message middleware deliver data to the corresponding consumers?

When designing producers and consumers, there is another important question to consider: what consumption pattern do we adopt when designing message middleware? Will the data be evenly distributed to consumers? Or will data be delivered to consumers through some other rules?

Support for distributed architecture

If we design message middleware, it will carry TB-level data high concurrency and high throughput write operations every day. Here, we need to consider designing message middleware as a distributed architecture.

When designing a distributed architecture, we also need to consider making the stored large data into sliced storage, slicing the data and other operations.

In addition to these, we need to consider another core issue: for message middleware, we need to support automatic expansion operations.

There is whether to support data fragmentation, how to achieve data fragmentation expansion and automatic data load balancing migration and so on.

High availability of data

The general high availability of Internet applications is achieved through local heap memory, distributed caching, and a copy of data on different servers. At this point, the downtime of any storage node will not affect the overall high availability. We can also refer to this idea when designing message middleware.

Message data is not lost

At this point, we need to provide a manual ACK mechanism, that is, when the consumer actually consumes the message, the message middleware returns the identity of "processing complete" to the message middleware, and the message middleware deletes the corresponding processed message.

However, to refine, here, we need two sets of ACK mechanisms:

One ACK corresponds to the production side. If the ACK message has not been received, you need to resend a message through the producer to ensure the success of the production message.

The other ACK corresponds to the consumer side. Once a message is consumed and processed successfully, an ack must be returned to the message middleware before the message middleware can delete the message. Otherwise, once the consumer goes down, you must resend the message to other consumer instances to ensure that the message will be processed successfully.

This is the end of the content of "how to design a highly concurrent message middleware". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.