Shulou(Shulou.com)06/01 Report--

This article mainly explains "what is the difference between RabbitMQ and Kafka". The explanation in this article is simple and clear, easy to learn and understand. Please follow the ideas of Xiaobian and go deep into it slowly to study and learn "what is the difference between RabbitMQ and Kafka" together.

# RabbitMQ Architecture ##Concepts

RabbitMQ is a distributed system.

**broker**: A service program run by each node, whose function is to maintain the addition and deletion of queues of the node and forward queue operation requests.

**master queue**: Each queue is divided into a main queue and several mirror queues.

**mirror queue**: mirror queue, as a backup to the master queue. After the node where the master queue is located hangs, the system promotes the mirror queue to the master queue, which is responsible for processing the client queue operation request. Note that mirror queue is only a mirror and is not designed to bear the read and write pressure of the client.

As shown in the figure above, there are two nodes in the cluster, each node has a broker, each broker is responsible for the maintenance of the queue on the local machine, and borkers can communicate with each other. There are two queues A and B in the cluster, each divided into a master queue and a mirror queue. So how does the production and consumption on the queue come about?

##Queue Consumption

As shown above, there are two consumer queues A, which are connected to different machines in the cluster. Any node in a RabbitMQ cluster has meta-information about all queues in the cluster, so you can connect to any node in the cluster. The main difference is that some consumers are connected to the node where the master queue is located, and some are connected to non-master queue nodes. Because the mirror queue needs to be consistent with the master queue, it needs synchronization mechanism. Because of the consistency limitation, all read and write operations must be operated on the master queue (think about it, why read from the master queue? And database read and write separation is different.), Then the master node synchronizes the operation to the node where the mirror queue is located. Even if the consumer is connected to a node other than the master queue, the consumer's operations are routed to the node where the master queue is located, so that consumption can occur.

##Queue Production

The principle is the same as consumption, if connected to a non-master queue node, route past.

** Insufficient ** Performance bottleneck due to master queue single node, throughput is limited. Although Erlang is used internally to improve performance, it cannot get rid of fatal flaws in architectural design.

# Kafka

To be honest, Kafka I think is to see RabbitMQ this defect to design an improved version, the improvement point is: a queue of a single master into multiple masters, that is, a machine can not withstand qps, then I use multiple machines to carry qps, a queue of traffic evenly distributed on multiple machines is not OK? Note that there is no intersection of data between multiple masters, i.e. a message is sent either to this master queue or to another master queue. Each master queue is called a Partition in Kafka, i.e. a fragment. A queue has multiple primary shards, and each primary shard has several secondary shards as backups. The synchronization mechanism is similar to RabbitMQ.

As shown above, we omit the different queues and assume that there is only one queue on the cluster (called Topic in Kafka). Each producer randomly sends messages to the primary shard, which then synchronizes to the secondary shards.

When the queue is read, the concept of a virtual Group is created. Messages within a Topic will only be routed to a consumer within the same Group. Messages consumed by consumers in the same Group are different. Groups share a Topic, which looks like multiple copies of a queue. Therefore, in order to achieve multiple groups sharing a Topic data, Kafka does not delete the message immediately after consumption like RabbitMQ, but must configure the save date in the background, that is, only save the message of the most recent period of time, and delete the message from the disk beyond this time. This ensures that the Topic data is visible to all groups within a period of time (this feature makes Kafka very suitable for a company data bus). Queue reads are also read primary shards, and in order to optimize performance, consumers have a one-to-one correspondence with primary shards. If the number of consumers is greater than the number of shards, some consumers do not get messages. It can be seen that Kafka is definitely designed for high throughput. For example, if the number of fragments is set to 100, then there will be 100 machines to carry the traffic of a Topic, which is of course better than RabbitMQ's stand-alone performance.

Thank you for reading, the above is "RabbitMQ and Kafka what is different" content, after the study of this article, I believe that we have a deeper understanding of RabbitMQ and Kafka what is different, the specific use of the situation also needs to be verified by practice. Here is, Xiaobian will push more articles related to knowledge points for everyone, welcome to pay attention!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.