Shulou(Shulou.com)06/03 Report--

This article mainly explains "what are the differences between the major mainstream message middleware". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Next let the editor to take you to learn "what are the differences between the major mainstream message middleware"!

Catalogue

Preface

1. Introduction of mainstream messaging Middleware-- ActiveMQ

1.1 Features

1.2 Architectural mode

1.3 Summary

two。 Introduction of mainstream messaging Middleware-- Kafka

2.1 Features

2.2 Architectural mode

2.3 Summary

3. Introduction of mainstream messaging Middleware-- RocketMQ

3.1 Features

3.2 Architectural mode

3.3 Summary

4. Why choose RabbitMQ?

5. Introduction of mainstream messaging Middleware-- RabbitMQ

5.1 Features

6. Comparative analysis diagram

Preface

Message queue has gradually become the core means of internal communication in enterprise IT system. It has a series of functions, such as low coupling, reliable delivery, broadcast, flow control, final consistency and so on. It has become one of the main means of asynchronous RPC. Nowadays, there are many mainstream news middleware in the market, such as the old ActiveMQ and RabbitMQ, the hot Kafka, the independent development of RocketMQ by Alibaba and so on. Today, we mainly introduce the differences and relations of the next several mainstream message middleware.

1. Introduction of mainstream messaging Middleware-- ActiveMQ

ActiveMQ is produced by Apache, and ActiveMQ is a JMS Provider implementation that fully supports JMS1.1 and J2EE 1.4 specifications. It is very fast, supports multiple language clients and protocols, and can be easily embedded in the enterprise application environment, and has many advanced features.

1.1 Features

ActiveMQ is the most popular and powerful open source message bus produced by Apache, and it is a message middleware that fully supports JMS specification.

Its rich API and a variety of cluster construction patterns make it a veteran message middleware in the industry, widely used in small and medium-sized enterprises!

MQ metrics: service performance, data storage, cluster architecture.

ActiveMQ is less used now because the performance of ActiveMQ is more mediocre than other MQ. Nowadays, the application scenarios of high concurrency and big data can be seen everywhere. If at this time in the choice of MQ, then ActiveMQ seems to be inadequate.

There are three main indicators to measure a MQ: service performance, data storage, and cluster architecture.

Service performance: the performance of ActiveMQ is not very good. In the face of large-scale concurrency, there will always be a variety of minor problems, such as blocking, excessive accumulation of messages, latency and so on.

Data storage: ActiveMQ uses KahaDB memory storage by default. You can also use some high-performance storage methods, such as: google's LevelDb is based on internal c memory. If it is to ensure the reliability of the message, you can also use mysql or oracle database.

Cluster architecture: ActiveMQ has been popular for so many years, and Api integrated with other components is also very complete. If not a very large concurrency scenario, ActiveMQ is also a good choice. Because ActiveMQ's cluster architecture model is also very good.

1.2 Architectural mode

Masrer-Slave mode

Active and standby mode, using Zookeeper to coordinate two or more nodes. Among them, the master node provides services, while the other slave nodes start, but do not provide services. When the master node dies, use Zookeeper to make a highly available handover, switch the Salve node to the master node, and continue to provide services.

NetWork mode

In essence, it is the integration of two groups of active and standby modes, and a distributed cluster can be realized by using a NewWork gateway and a connection configuration.

1.3 Summary

Advantages:

Cross-platform (JAVA writing is platform independent, ActiveMQ can run on almost any JVM)

You can use JDBC: you can persist data to a database. Although using JDBC degrades the performance of ActiveMQ, the database has always been the most familiar storage medium for developers.

Support JMS specification: support the unified interface provided by JMS specification

Support for automatic reconnection and error retry mechanism

Security mechanism: supports multiple security configuration mechanisms based on shiro,jaas, and can authenticate and authorize Queue/Topic

Perfect monitoring: complete monitoring, including WebConsole,JMX,Shell command line, Jolokia RESTful API

User-friendly: the WebConsole provided can satisfy most situations, and there are many third-party components that can be used, such as hawtio

Disadvantages:

Community activity is not as high as RabbitMQ.

According to other users' feedback, there will be inexplicable problems and messages will be lost.

At present, the focus is on the activemq6.0 product Apollo, with less maintenance for 5.x.

Not suitable for application scenarios with thousands of queues

two。 Introduction of mainstream messaging Middleware-- Kafka

Apache Kafka is a distributed message publishing and subscribing system. It was originally implemented as a distributed log submission system (a distributed commit log) by LinkedIn based on its unique design, and later became part of the Apache project. Kafka is efficient, scalable, and persistent. Its partition feature, replicability and fault tolerance are all good features.

2.1 Features

Kafka is LinkedIn's open source distributed distribution-scheduled messaging system that currently belongs to the Apache top-level project. The main feature of Kafka is to give Pull a mode to handle consumer messages in pursuit of high throughput, which is intended for log collection and transmission from the very beginning. Version 0.8 starts to support replication, does not support transactions, and has no strict requirements for message repetition, loss and errors, so it is suitable for the data collection business of Internet services that generate a large amount of data. You can see here that kafka is only concerned with throughput. Therefore, when using kafka, pay attention to whether the business allows message duplication, loss, errors, and so on. If allowed, kafka is the most appropriate. Because its performance is the highest. Even on cheap servers, it can support more than 100k pieces of data per second on a single machine. So its performance is very good. Kafka only uses memory for storage, as long as there is enough memory, it can be large enough throughput. Because kafka does not read or write on disk.

Fast persistence: message persistence can be performed at O (1) overhead

High throughput: it can reach the throughput rate of 10W/s on an ordinary server

Fully distributed systems: Broker, Producer and Consumer all support distribution automatically and realize load balancing automatically.

Supports two highly available mechanisms: synchronous and asynchronous replication

Support batch sending and pulling of data

Zero copy technology (zero-copy): reduce IO operation steps and improve system throughput

Data migration and expansion are transparent to users

The machine can be expanded without downtime

Other features: rich message pull model, efficient subscriber level expansion, real-time message subscription, 100 million-level message stacking capability, periodic deletion mechanism

2.2 Architectural mode

Kafka architecture model

Mainly rely on Zookeeper for coordination management, each kafka can carry out replica replication, that is, data synchronization. If you say: a piece of data falls on the first node, then repilicate replication will be carried out, so that each node will have a piece of data in operation, with a total of three points of data. If one of them is down, you can also get data from the other two nodes. Deployment proposal: deploy across computer rooms. Even if one machine goes down, there is no problem with the data. If there is a downtime in the whole location. Then our data will be lost. This is also a remote disaster preparedness that large companies need to consider. Of course, kafka is mainly concerned with performance and high reliability of data.

2.3 Summary

Advantages:

Rich client languages: support Java, .net, PHP, Ruby, Python, Go and other languages

High performance: write TPS on a single machine to about 1 million messages per second, message size 10 bytes

Fully distributed architecture, replica mechanism, high availability and reliability, and theoretically support unlimited accumulation of messages

Batch operation is supported

Consumers use Pull to get the message. Messages are orderly, and all messages can be consumed and consumed only once through control.

Excellent third-party KafkaWeb management interface Kafka-Manager

It is more mature in the field of logging and is used by many companies and open source projects.

Disadvantages:

When there are more than 64 queues / partitions on a single Kafka, there will be a sharp rise in Load. The more queues, the higher the load, and the longer the response time for sending messages

Using short polling mode, real-time performance depends on the polling interval.

Consumption failure does not support retry

Message order is supported, but when an agent goes down, messages are out of order.

Community updates are slow.

3. Introduction of mainstream messaging Middleware-- RocketMQ

RocketMQ is Ali's open source messaging middleware, and it has also hatched into a top-level Apache project. Implemented in the Java language, referred to Kafka in the design, and made some of its own improvements, the message reliability is better than Kafka. RocketMQ is widely used in Ali internal scenarios such as order, transaction, recharge, flow computing, message push, log streaming, binglog distribution and so on.

3.1 Features

The core features are as follows:

Ensure the sequence of messages, and messages are consumed sequentially.

Provides a wealth of pull and processing modes.

Efficient subscribers can also scale horizontally.

It carries hundreds of millions of levels of message stacking capacity.

3.2 Architectural mode

RocketMQ cluster architecture model

1.Master-Slave (master-slave) mode

two。 Dual Master mode.

3. Double master and double slave mode.

4. Multi-master and multi-slave mode.

5. One master and multiple slaves mode.

There are many options to choose from.

And so on, with reference to many open source settings.

Cluster topology

Ali felt that the performance of Zookeeper was too low, so he built NameServer himself, and the NameServer code was also very concise, with only a few hundred lines of code. You can read the source code if you are interested.

3.3 Summary

Advantages:

A single machine supports more than 10,000 persistence queues.

All RocketMQ messages are persistent. Write to the system PAGECACHE first, and then flush the disk to ensure that there is a copy of data in memory and disk, and when accessed, it is read directly from memory.

The model is simple and the interface is easy to use (the interface of JMS is not very practical in many cases)

The performance is very good, which allows a large number of messages to be stacked in Broker

Support a variety of consumption patterns, including cluster consumption, broadcast consumption, etc.

Distributed expansion design for all links, supporting master-slave and high availability

The degree of development is more active, and the version is updated quickly.

Disadvantages:

There are not many client languages supported. At present, they are Java and Clippers, among which C++ is not yet mature.

Maintenance of RocketMQ requires a professional team

For the commercial version, there are many functions that are not available to the public.

Interfaces such as JMS are not implemented in the MQ core.

4. Why choose RabbitMQ?

1.ActiveMQ, the performance is not very good, so in the high concurrency scenario, it is directly dropped by pass. Its Api is perfect and can be used by small and medium-sized Internet companies.

2.kafka, which mainly emphasizes high performance, when the delivery of reliable messages is required for the business. Then you can't choose kafka. But if you do some log collection, kafka is still very good. Because the performance of kafka is very good.

3.RocketMQ, its characteristics are very good. It has high performance, reliability, distributed things, support for horizontal expansion, hundreds of millions of levels of message accumulation, master-slave switching and so on. It basically satisfies all the advantages of MQ. But its biggest disadvantage: charging for the commercial version. Therefore, it has many functions that are not provided to the outside world.

So after talking about these three MQ, are there any other MQ to choose from? Yes, it is also the MQ--RabbitMQ of this study.

5. Introduction of mainstream messaging Middleware-- RabbitMQ

RabbitMQ, released in 2007, is a reusable enterprise messaging system based on AMQP (Advanced message queuing Protocol). It is one of the most mainstream message middleware at present.

5.1 Features

RabbitMQ is an open source message queuing system developed in Erlang language and implemented based on AMQP protocol.

The main features of AMQP are message-oriented, queue-oriented, routing (including peer-to-peer and publish / subscribe), reliability and security.

AMQP protocol is more used in enterprise systems, requiring high data consistency, stability and reliability, followed by performance and throughput requirements.

The reliability of RabbitMQ is very good, and the data can be guaranteed 100% not to be lost. Mirror queues can be used, and its stability is very good. So in our Internet financial industry. When the requirements for the stability and reliability of the data are very high, we will choose RabbitMQ. Certainly not as good as kafka, but much better than AvtiveMQ. You can also do some performance optimization on your own.

RabbitMQ can build a remote dual-live architecture, including that each node can be stored in disk or memory mode.

Cluster architecture of RabbitMQ

What the figure says is that we can use three nodes as a set of clusters for RabbitMQ, and of course there can be many groups. Mirror queue is used between nodes. Based on this way, 100% data can be guaranteed not to be lost.

The front end can do load balancing, such as load balancing component: HA-proxy, to carry out load at TCP level.

If you want to make a high availability, you need to make a high availability configuration with the help of keepAlived.

For example, a virtual VIP is added to the front end, which is routed to the specified load balancer component through VIP, and then to a node in the RabbtMQ.

This is the entire RabbitMQ cluster architecture.

Can achieve very perfect, high availability and performance is also very good, super stability. And there are various means of cluster recovery.

For example, if a node is down, or a disk is damaged, it can also perform a message repair. Based on so many advantages, we must learn RabbitMQ well.

6. Comparative analysis diagram

At this point, I believe you have a deeper understanding of "what are the differences between the major mainstream message middleware". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.