Shulou(Shulou.com)05/31 Report--

This article introduces you how to ensure the high availability of the RabbitMQ message queue, the content is very detailed, interested friends can refer to, hope to be helpful to you.

Interview questions

How to ensure the high availability of message queues?

Psychological analysis of interviewer

If someone asks about your knowledge of MQ, high availability is a must. MQ can reduce the availability of the system. So as long as you use MQ, some of the next questions must be around how to solve the shortcomings of MQ.

If you are stupid enough to use a MQ and never think about all kinds of questions, you will be miserable. The interviewer's impression of you is that he will simply use some technology without any thinking, and he or she will not have a good impression of you right away. If such a student is recruited to be an ordinary younger brother with a salary of less than 20k, it will be miserable if he is a senior worker with a salary of 20k +. If you design a system, there must be a pile of pits in it, and the company will suffer losses in an accident, and the team will carry the pot together.

Analysis of interview questions

It's a good question to ask, because I can't ask you how to guarantee the high availability of Kafka. How to ensure the high availability of ActiveMQ? If an interviewer asks such a question, it looks very unskillful. People may use RabbitMQ, but they have not used Kafka. Why do you come up and ask others about Kafka? I thought it was clear that it was difficult for others.

So the competent interviewer is asking how to guarantee the high availability of MQ? This is which MQ you have used, so tell me your understanding of the high availability of that MQ.

High availability of RabbitMQ

RabbitMQ is representative because it is based on master-slave (non-distributed) high availability, so let's take RabbitMQ as an example to explain how to implement the high availability of the first kind of MQ.

There are three modes of RabbitMQ: stand-alone mode, ordinary cluster mode and mirror cluster mode.

Stand-alone mode

Stand-alone mode, which is at the Demo level, is generally started by you to play locally, and no one produces it.

Normal cluster mode (no high availability)

Normal cluster mode means that multiple RabbitMQ instances are started on multiple machines, one for each machine. The queue you create will only be placed on one instance of RabbitMQ, but each instance synchronizes the metadata of queue (metadata can be thought of as some configuration information of queue, through which you can find the instance of queue). When you consume, in fact, if you connect to another instance, that instance will pull data from the instance where the queue is located.

This approach is really troublesome, and it is not very good. Without the so-called distributed approach, it is just an ordinary cluster. Because this causes either consumers to randomly connect to one instance at a time and pull data, or to permanently connect to the instance where the queue is located to consume data. The former has the overhead of data pull, while the latter leads to the performance bottleneck of a single instance.

Moreover, if the instance that put queue goes down, other instances cannot be pulled from that instance. If you enable message persistence and let RabbitMQ store messages on the ground, messages may not be lost. You have to wait for the instance to be restored before you can continue to pull data from this queue.

So this is rather awkward, there is no so-called high availability, this solution is mainly to improve throughput, that is, let multiple nodes in the cluster to serve the read and write operations of a queue.

Mirror cluster mode (high availability)

This mode is the so-called high availability mode of RabbitMQ. Unlike the normal cluster mode, in the image cluster mode, messages in both metadata and queue of the queue you create will exist on multiple instances, that is, each RabbitMQ node has a complete image of the queue, which means all the data of the queue. Then every time you write a message to queue, it will automatically synchronize the message to the queue of multiple instances.

File

So how to turn on this mirror cluster mode? In fact, it is very simple. RabbitMQ has a good management console, that is, a new policy is added in the backend. This policy is the policy of mirror cluster mode. When you specify, you can require data synchronization to all nodes or to a specified number of nodes. When you create a queue again, apply this policy, and the data will be automatically synchronized to other nodes.

In this case, the advantage is that any of your machines are down, it's okay, other machines (nodes) also contain the complete data of this queue, and other consumer can consume data on other nodes. The downside is that, first, this performance overhead is too high, messages need to be synchronized to all machines, resulting in heavy network bandwidth pressure and consumption! Second, play like this, if it is not distributed, there will be no scalability at all. If a queue is heavily loaded and you add a machine, the new machine also contains all the data of the queue, and there is no way to linearly expand your queue. Do you think, what if the amount of data in this queue is so large that the capacity on this machine is too large to hold?

High availability of Kafka

Kafka one of the most basic architectural understanding: consists of multiple broker, each broker is a node; you create a topic, this topic can be divided into multiple partition, each partition can exist on a different broker, each partition will put a part of the data.

This is a natural distributed message queue, that is, the data of an topic is distributed on multiple machines, and each machine puts part of the data.

In fact, things like RabbmitMQ are not distributed message queues, they are traditional message queues, but they only provide some clustering and HA (High Availability, high availability) mechanisms, because no matter how much you play, the data of a RabbitMQ queue is placed in a node, and under the mirror cluster, each node puts the complete data of the queue.

Before Kafka 0.8, there was no HA mechanism, that is, if any broker went down, the partition on that broker was invalidated, unable to write or read, and there was no high availability to speak of.

For example, let's say we create a topic and specify that the number of partition is three, each on three machines. However, if the second machine goes down, it will cause the data of this topic to be lost, so this will not be highly available.

File

After Kafka 0.8, the HA mechanism is provided, that is, the replica (replica) copy mechanism. The data of each partition is synchronized to other machines to form its own multiple replica copies. All replica will elect a leader, then production and consumption will deal with this leader, and then the other replica is follower. When writing, leader will be responsible for synchronizing the data to all follower, and when reading, you can just read the data on the leader. Can only read and write leader? Quite simply, if you can read and write each follower at will, then you will have the problem of care data consistency. The complexity of the system is too high and it is easy to cause problems. Kafka evenly distributes all the replica of a partition on different machines, which improves fault tolerance.

File

In this way, there is the so-called high availability, because if a broker goes down, it's okay, the partition on that broker has copies on other machines. If there is a partition leader on the down broker, a new leader will be re-elected from the follower at this time, and everyone can continue to read and write that new leader. This is called high availability.

When writing data, the producer writes leader, then leader writes the data to the local disk, and then other follower actively pull the data from leader. Once all the follower has synchronized the data, the ack will be sent to the leader,leader to receive all the follower's ack, and the successful write message will be returned to the producer. (of course, this is just one of the patterns, and the behavior can be adjusted appropriately)

When consuming, it will only be read from leader, but only if a message has been successfully returned to ack by all follower synchronously will the message be read by the consumer.

On how to ensure the high availability of RabbitMQ message queues to share here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.