Shulou(Shulou.com)06/02 Report--

This article will explain in detail what is the design scheme of the Broker master-slave architecture of message queuing. The content of the article is of high quality, so the editor will share it with you for reference. I hope you will have some understanding of the relevant knowledge after reading this article.

Today we are going to learn about the underlying module of message queue design, the architecture design of Broker.

How to synchronize Master Broker and Slave Broker messages

As we know before, if we want Broker to support high availability, we should design it as a master-slave architecture. The previous distributed storage has also talked a lot about this architecture, so you can check the history article yourself.

First of all, let's look at the first problem. In order to ensure that the data in our MQ is not lost and that it is available, we design Broker as a Master-Slave pattern, that is, a Master Broker corresponds to multiple Slave Broker.

The advantage is that when our Master Broker receives the message, it synchronizes the message to Slave, so even if the Master Broker goes down, there is still data on the Slave.

As above, let's consider, how does this Master Broker synchronize data to Slave Broker? There are generally two options:

Master Broker actively pushes messages to Slave Broker.

Slave Broker sends a request to Master Broker to pull message data.

We use the second pull scheme, which is more reliable, and let Slave Broker keep sending requests to Master Broker to achieve pull mode to pull messages.

Does MQ achieve read-write separation?

As we already know from the above, Master Broker is mainly used to receive messages and then synchronize to Slave Broker, so Slave Broker has exactly the same data.

In that case, our next question is, does the consumer system get information from Master Broker or from Slave Broker?

In fact, we can't simply get it from master or slave. We should be smarter. It's possible to get it from either Master or Slave.

As a consumer system, when getting messages, the request will first be sent to Master Broker, and then Master Broker will return a batch of messages to the consumer system.

Then, when Master Broker returns the message to the consumer system, it will suggest to the consumer system whether to get the message from Master Broker or Slave Broker next time according to its own load and synchronization with Slave.

For example, now that Master has a heavy load, it has to resist 100000 write and concurrency, and then you have to get messages from it, which will put a heavier burden on Master, so Master Broker will recommend that you go to Slave Broker to pull messages.

For example, now that Master Broker has received 1 million messages, the Slave Broker machine does not know why it synchronizes to 960000 messages, lagging behind 40,000 message data. at this time, the consumer system may have obtained 960000 messages, so the next time you can only pull messages from Master. Because Slave Broker synchronization messages are too slow, we can't get the latest news from there.

So, it's up to Master Broker to decide where to get the message based on the actual load.

As shown in the figure:

When writing a message, it is generally certain that Master Broker is selected to write.

When consuming messages, it is possible to pull messages in Master Broker or in Slave Broker, depending on the situation at that time.

What's the impact of Slave Broker hanging up?

Now let's look at the next question, if Slave Broker dies, what will be the impact on our entire system? The impact is a little bit, but not too big, fearless.

Because messages are all sent to Master Broker when they are written, and then you can also go to Master Broker when pulling messages, but some messages may be pulled on Slave Broker.

Therefore, if the Slave Broker is down, our message writing and access can go to Master Broker, which will not have a great impact on our overall system. That is, it may increase the pressure on Master Broker to read and write.

What should I do if Master Broker is dead?

Above we analyzed that the failure of Slave Broker does not affect the overall system, and now suppose our Master Brokker is freaked out and hung up, what will happen?

At this time, it will have a certain impact on the writing and acquisition of messages, but in essence, there is a piece of data on Slave Broker, but there is no time for some data to be synchronized from Master Broker. Generally speaking, we have to design Slave Broker to automatically take over Master Broker mechanism. There are two solutions:

Manual operation and maintenance, switched by human hand

Use tools to switch automatically

Manual switching

Before the RocketMQ4.5 version, manual OPS is always used. When the Master Broker is hung up, the configuration is artificially modified, the Slave Broker is modified, and then the restart of the machine is adjusted to Master Broker, which is a bit troublesome during the period, and will be unavailable for a short time.

Using the above approach, we can not completely achieve high availability, because there is no way to automatically upgrade Slave Broker to Master Broker.

Automatic switching of MQ based on Dledger

After RocketMQ4.5, we begin to introduce a new mechanism, that is, Dledger,Dledger is implemented based on the Raft protocol, and its underlying principles will be explained in detail later. Let's first look at how to achieve automatic switching based on Dledger.

After RocketMQ introduces Dledger, you can make one Master Broker correspond to multiple Slave Broker, that is, there will be multiple copies of one data. For example, we have one Master Broker corresponding to two Slave Broker.

At this point, if a Master Broker goes down, there will still be multiple Slave, and then the leader will be selected through Dledger technology and Raft protocol. In fact, I have a special article on the master algorithm, which can be seen (whether the interview is often asked about the core questions of the distributed system, no one beats you this time). In this way, a new Master broker will be selected to provide services.

In this way, the whole process will be very fast, about ten seconds or dozens of seconds to complete the switching action, completely and automatically select Slave Broker as the Master broker external service to achieve high availability mode.

On the message queue Broker master-slave architecture design is shared here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.