Shulou(Shulou.com)06/01 Report--

Editor to share with you how to choose RabbitMQ and Kafka, I believe that most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to know it!

RabbitMQ architecture

RabbitMQ is a distributed system in which there are several abstract concepts.

Broker: the service program that each node runs to maintain the addition or deletion of the queue of that node and to forward queue operation requests.

Master queue: each queue is divided into a primary queue and several mirror queues.

Mirror queue: mirror queue as a backup of master queue. After the master queue node dies, the system promotes mirror queue to master queue, which is responsible for handling client queue operation requests. Note that mirror queue is only mirrored and is not designed to bear the pressure of reading and writing on the client.

As shown in the figure above, there are two nodes in the cluster, and each node has a broker, each broker is responsible for the maintenance of queues on the local machine, and the borker can communicate with each other.

There are two queues An and B in the cluster, each divided into master queue and mirror queue (backup). So how is the production and consumption on the queue realized?

Queue consumption

For example, in the figure above, there are two consumer consumption queues A, which are connected to different machines in the cluster. Any node in the RabbitMQ cluster has the meta-information of all queues on the cluster, so it is possible to connect to any node in the cluster. The main difference is that some consumer is connected to the node where the master queue is located, and some are connected to the non-master queue node.

Because mirror queue is to be consistent with master queue, a synchronization mechanism is needed. Because of the limitation of consistency, all read and write operations must be operated on master queue. (think about it, why read from master queue? It is different from the separation of database read and write. ),

Then the master node synchronizes to the node where the mirror queue is located.

Even if the consumer is connected to a non-master queue node, the operation of the consumer is routed to the node where the master queue is located so that it can be consumed.

Queue production

The principle is the same as consumption: if you connect to a non-master queue node, you will route the past.

Therefore, friends can see the shortcomings of RabbitMQ: due to the master queue single node, resulting in performance bottlenecks, throughput constraints. Although Erlang is implemented internally in order to improve performance, it can not get rid of the fatal flaw in architecture design.

Kafka

To tell you the truth, Kafka, I think it is an improved version designed because of the defect of RabbitMQ. The improvement point is: change a single master in a queue into multiple master, that is, a machine cannot carry qps, so I can use multiple machines to carry qps, and distribute the traffic of a queue evenly across multiple machines.

Note that there is no intersection of data between multiple master, that is, a message is either sent to this master queue or to another master queue.

Each master queue in this is called a Partition in Kafka, that is, a fragment. A queue has multiple primary shards, and each primary shard has several secondary shards for backup. The synchronization mechanism is similar to RabbitMQ.

As shown in the figure above, we omit different queue and assume that there is only one queue (Topic in Kafka) on the cluster. Each producer randomly sends the message to the primary shard, which is then synchronized to the secondary shard.

The concept of a Group is simulated when the queue is read. A message within a Topic will only be routed to a consumer in the same Group. The messages consumed by the consumer in the same Group are different. A Topic is shared between the Group, which seems to be multiple copies of a queue.

Therefore, in order to achieve multiple Group sharing a Topic data, Kafka will not delete messages like RabbitMQ immediately after consumption, but must configure the save date in the background, that is, only the most recent messages will be saved, and messages beyond this period of time will be deleted from disk, thus ensuring that Topic data is visible to all Group within a period of time (this feature makes Kafka very suitable for a company's data bus).

Queue reading also reads the main shard, and in order to optimize the performance, there is an one-to-one correspondence between the consumer and the main shard. If the number of consumers is greater than the number of shards, there are some consumers who can not get the message.

Thus it can be seen that Kafka is definitely designed for high throughput. For example, if the number of shards is set to 100, then there are 100 machines to carry the traffic of one Topic, which is certainly better than the stand-alone performance of RabbitMQ.

This article only makes a comparison between Kafka and RabbitMQ, but not only these two open source queues, ZeroMQ,RocketMQ,JMQ, and so on, there is no time to take a closer look, so it is not within the scope of this article.

So, don't be confused by these various queues, find out the key differences from the architecture, and choose the model easily according to your actual needs (for example, this article only examines the throughput requirement).

Finally, it is summarized as follows:

Low throughput: both Kafka and RabbitMQ are fine.

High throughput: Kafka.

The above is all the content of the article "how to choose RabbitMQ and Kafka". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.