Shulou(Shulou.com)06/02 Report--

How to compare the four MQ of message middleware? I believe many inexperienced people are at a loss about this. Therefore, this paper summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

1. Overview

Message queue has gradually become the core means of internal communication in enterprise IT system. It has a series of functions, such as low coupling, reliable delivery, broadcast, flow control, final consistency and so on. It has become one of the main means of asynchronous RPC. Nowadays, there are many mainstream news middleware in the market, such as the old ActiveMQ and RabbitMQ, the hot Kafka, the independent development of RocketMQ by Alibaba and so on.

2. The composition of message middleware

2.1 Broker

Message server, which provides core message service as server

2.2 Producer

The message producer, the initiator of the service, is responsible for the production and transmission of messages to broker

2.3 Consumer

The message consumer, the processor of the business, is responsible for obtaining the message from the broker and processing the business logic.

2.4 Topic

Topic, publish and subscribe messages under the unified gathering place, different producers send messages to topic, and MQ servers distribute them to different subscribers to realize message broadcasting.

2.5 Queue

Queue, in PTP mode, specific producers send messages to specific queue, and consumers subscribe to specific queue to receive specified messages.

2.6 Message

The message body, which is encoded according to the fixed format defined by different communication protocols, encapsulates the business data and transmits the message.

3. Message middleware pattern classification

3.1 Point-to-Point

PTP peer-to-peer: using queue as a communication carrier

Description:

The message producer produces the message to the queue, and then the message consumer takes the message from the queue and consumes the message.

After the message is consumed, it is no longer stored in the queue, so it is impossible for the message consumer to consume the message that has been consumed. Queue supports multiple consumers, but for a message, only one consumer can consume.

3.2 publish / subscribe

Pub/Sub publish and subscribe (broadcast): use topic as the communication carrier

Description:

The message producer (publisher) publishes the message to topic, and multiple message consumers (subscribers) consume the message. Unlike peer-to-peer, messages posted to topic are consumed by all subscribers.

Queue implements load balancing, sending messages produced by producer to message queues, which are consumed by multiple consumers. However, a message can only be accepted by one consumer, and when no consumer is available, the message is saved until a consumer is available.

Topic implements publish and subscribe, and when you publish a message, all services that subscribe to the topic can get the message, so from 1 to N subscribers can get a copy of the message.

4. Advantages of message middleware

4.1 system decoupling

There is no direct call relationship between interactive systems, only through message transmission, so the system invasion is not strong, the degree of coupling is low.

4.2 improve system response time

For example, in the original set of logic, the completion of payment may involve modifying the status of the order, calculating membership points, and notifying logistics and distribution before it can be completed; through the MQ architecture design, the urgent and important business (requiring immediate response) can be put into the call method, and the message queue with low response requirements can be put into the MQ queue for consumers to process.

Provide services for big data's processing architecture

Through message as integration, under the background of big data, message queue is also integrated with real-time processing architecture to provide performance support for data processing.

4.4 Java message Service-JMS

Java message Service (Java Message Service,JMS) application program interface (API) is an API about message oriented middleware (MOM) in a Java platform, which is used to send messages between two applications or in distributed systems for asynchronous communication.

P2P and Pub/Sub message patterns in JMS: peer-to-peer (point to point, queue) and publish subscription (publish/subscribe,topic) were originally defined by JMS. The main difference or solution between the two modes is whether messages sent to the queue can be repeatedly consumed (multiple subscriptions).

5. Application scenario of message middleware

5.1 Asynchronous Communication

Some businesses do not want or need to process messages immediately. Message queuing provides an asynchronous processing mechanism that allows users to put a message on the queue, but does not process it immediately. Put as many messages as you want in the queue, and then process them when needed.

5.2 decoupling

Reduce the degree of strong dependence between projects and adapt to heterogeneous systems. It is extremely difficult to predict what requirements the project will encounter in the future at the beginning of the project. Through the message system, an implicit data-based interface layer is inserted in the middle of the processing process, and both sides of the processing process should implement this interface. When the application changes, the processing processes of both sides can be extended or modified independently. Just make sure they follow the same interface constraints.

5.3 redundancy

In some cases, the process of processing data will fail. Unless the data is persisted, it will be lost. Message queues persist data until they have been fully processed, avoiding the risk of data loss. In the insert-get-delete paradigm used by many message queues, before deleting a message from the queue, your processing system needs to clearly indicate that the message has been processed to ensure that your data is safely saved until you have finished using it.

5.4 scalability

Because message queues decouple your processing, it is easy to increase the frequency of queuing and processing of messages, as long as you add additional processing. There is no need to change the code or adjust the parameters. It is convenient for distributed expansion.

5.5 overload protection

In the case of a sharp increase in traffic, applications still need to continue to play a role, but such sudden traffic can not be extracted and predicted; if you think that you can handle this kind of instantaneous peak access as the standard to invest resources on standby is undoubtedly a huge waste. The use of message queuing enables key components to withstand sudden access pressure without completely collapsing due to sudden overloaded requests.

5.6 recoverability

When some components of the system fail, the whole system will not be affected. Message queuing reduces the coupling between processes, so even if a process that processes messages dies, messages added to the queue can still be processed after the system is restored.

5.7 Sequential guarantee

In most usage scenarios, the order of data processing is important. Most message queues are sorted by nature, and it is guaranteed that the data will be processed in a specific order.

5.8 buffer

In any important system, there will be elements that require different processing times. Message queuing facilitates the most efficient execution of tasks through a buffer layer that helps control and optimize the speed at which data flows through the system. To adjust the system response time.

5.9 data stream processing

The massive data streams generated by distributed systems, such as business logs, monitoring data, user behavior, etc., are collected in real time or in batches, and then analyzed by big data is a necessary technology on the Internet. it is the best choice to collect this kind of data through message queue.

6. Common protocols of message middleware

6.1 AMQP protocol

AMQP, namely Advanced Message Queuing Protocol, is an application layer standard advanced message queuing protocol that provides unified messaging services. It is an open standard of application layer protocols and is designed for message-oriented middleware. The client and message middleware based on this protocol can transmit messages and are not limited by different client / middleware products, different development languages and other conditions.

Advantages: reliable and universal

6.2 MQTT protocol

MQTT (Message Queuing Telemetry Transport, message queuing Telemetry Transmission) is an instant messaging protocol developed by IBM, which may become an important part of the Internet of things. The protocol supports all platforms, connects almost all connected objects to the outside, and is used as a communication protocol for sensors and actuators, such as connecting houses through Twitter.

Advantages: simple format, small bandwidth consumption, mobile communication, PUSH, embedded system

6.3 STOMP protocol

STOMP (Streaming Text Orientated Message Protocol) is a streaming text oriented message protocol, which is a simple text protocol designed for MOM (Message Oriented Middleware, message-oriented middleware). STOMP provides an interoperable connection format that allows clients to interact with any STOMP message broker (Broker).

Advantages: command mode (non-topic\ queue mode)

6.4 XMPP protocol

XMPP (Extensible message processing Field Protocol, Extensible Messaging and Presence Protocol) is a protocol based on Extensible markup language (XML), which is mostly used for instant messaging (IM) and online live detection. It is suitable for quasi-instant operation between servers. The core is based on XML streaming, a protocol that could eventually allow Internet users to send instant messages to anyone else on the Internet, even if the operating system and browser are different.

Advantages: universal openness, strong compatibility, scalability and high security, but XML coding format occupies a large bandwidth.

6.5 other protocols based on TCP/IP customization

Some special frameworks (such as redis, kafka, zeroMq, etc.) do not strictly follow the MQ specification according to their own needs, but encapsulate a set of protocols based on TCP\ IP and transmit through the network socket interface to realize the function of MQ.

7. Introduction of common message middleware MQ

7.1 RocketMQ

Ali is an open source distributed, queuing model message middleware, the original name of Metaq,3.0 version is changed to RocketMQ, which is a set of mq implemented by Ali with java according to the design idea of kafka. At the same time, a number of mq products (Notify, metaq) within the Ali system are integrated to maintain only the core functions, removing all other runtime dependencies, and ensuring the simplification of the core functions. On this basis, we cooperate with Ali's other open source products to achieve the architecture of mq in different scenarios, which is mainly used in the order trading system.

It has the following characteristics:

Can guarantee strict message order

Provide filtering for messages

Provide rich message pull mode

Efficient subscriber horizontal scalability

Real-time message subscription mechanism

100 million-level message accumulation capability

Officials provide some comparative differences that are different from kafka:

Https://rocketmq.apache.org/docs/motivation/

7.2 RabbitMQ

An open source message queue written in Erlang supports many protocols: AMQP,XMPP, SMTP,STOMP, which makes it very heavyweight and more suitable for enterprise development. At the same time, the Broker architecture is implemented, the core idea is that the producer will not send the message directly to the queue, and the message will be queued in the central queue when it is sent to the client. It has good support for routing (Routing), load balancing (Load balance) and data persistence. It is mostly used for enterprise-level ESB integration.

7.3 ActiveMQ

A subproject under Apache. With Java's JMS Provider implementation that fully supports JMS1.1 and J2EE 1.4 specifications, advanced application scenarios can be implemented efficiently with a small amount of code. Pluggable transport protocol support, such as: in-VM, TCP, SSL, NIO, UDP, multicast, JGroups and JXTA transports. RabbitMQ, ZeroMQ and ActiveMQ all support commonly used multilingual clients such as C++, Java, .net, Python, Php, Ruby and so on.

7.4 Redis

Use C language to develop a Key-Value NoSQL database, development and maintenance is very active, although it is a Key-Value database storage system, but it supports MQ function, so it can be used as a lightweight queue service. For the entry and exit operations of RabbitMQ and Redis, each performed 1 million times, and the execution time was recorded every 100000 times. The test data are divided into four different sizes: 128Bytes, 512Bytes, 1K and 10K. The experimental results show that when joining the team, the performance of Redis is higher than that of RabbitMQ when the data is compared, but it is unbearably slow if the data size exceeds 10K; when leaving the team, Redis shows very good performance regardless of the size of the data, while the performance of RabbitMQ is much lower than that of Redis.

7.5 Kafka

A subproject under Apache, a high-performance distributed Publish/Subscribe message queuing system implemented with scala, has the following characteristics:

Fast persistence: messages can be persisted at O (1) overhead through disk sequential read-write and zero-copy mechanism

High throughput: it can reach the throughput rate of 10W/s on an ordinary server

High accumulation: consumers are supported to be offline for a long time under topic, resulting in a large amount of messages.

Fully distributed systems: Broker, Producer and Consumer all support distribution automatically, and rely on zookeeper to automatically realize complex equalization.

Support for parallel loading of Hadoop data: for log data and offline analysis systems like Hadoop's, but with real-time processing limitations, this is a feasible solution.

7.6 ZeroMQ

The so-called fastest message queuing system, specially developed for high-throughput / low-latency scenarios, is often used in financial applications, focusing on real-time data communication scenarios. ZMQ can implement advanced / complex queues that RabbitMQ is not good at, but developers need to combine a variety of technical frameworks themselves, and the development cost is high. So ZeroMQ has a unique non-middleware pattern, more like a socket library, you do not need to install and run a message server or middleware, because your application itself is to use ZeroMQ API to complete the role of logical services. But ZeroMQ only provides non-persistent queues, and data will be lost if the downmachine is used. For example, ZeroMQ is used as the transmission of data stream in the Storm of Twitter.

ZeroMQ sockets are independent of the transport layer: ZeroMQ sockets define a unified API interface for all transport layer protocols. The default supports in-process (inproc), inter-process (IPC), multicast, TCP protocols, switching between different protocols as long as simply change the prefix of the connection string. You can switch from local inter-process communication to distributed TCP communication at any time with minimal cost. ZeroMQ handles connection establishment, disconnection, and reconnection logic behind the scenes.

Properties:

Lock-free queue model: for the data exchange channel pipe between inter-thread interactions (client and session), the lock-free queue algorithm CAS; is used to register asynchronous events at both ends of pipe. When reading or writing messages to pipe, read and write events are automatically triggered.

Batch processing algorithm: for batch messages, it is adaptively optimized to receive and send messages in batches.

Thread binding under multi-core does not need CPU switching: different from the traditional multi-thread concurrent mode, semaphore or critical area, zeroMQ makes full use of the advantages of multi-core, each core binding runs a worker thread to avoid the overhead of CPU switching between multi-threads.

After reading the above, have you mastered how to compare the four MQ of message middleware? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.