Shulou(Shulou.com)06/02 Report--

This article focuses on "how to design a MQ message queue from 0 to 1". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn how to design a MQ message queue from 0 to 1.

Overall Design idea of message queue

The main purpose is to design an overall data flow in which messages are consumed.

Here will involve: message production Producer, Broker (message server), message consumer Consumer.

1.Producer (message producer): sends messages to Broker.

2.Broker (server side): the concept of Broker mainly comes from the ActiveMQ of Apache, especially the server side of message queue.

The main function is to transmit the message from the sender to the receiver, which involves message storage, message communication mechanism and so on.

3.Consumer (message consumer): receives a message from the message queue, and the consumer replies to the consumption confirmation.

Broker (message queuing server) design focus

1) message dump: delivered at a more appropriate time, or through a series of means to assist the message can eventually be delivered to the consumer machine.

2) standardize a paradigm and a general model to meet the requirements of decoupling, final consistency, off-peak and so on.

3) in fact, it is simply a message repeater, which converts a RPC into two RPC, and the sender delivers the message to the broker,broker and then forwards the message to the receiver.

To sum up, it is two RPC plus one dump, and if you want to do consumption confirmation, it is three RPC.

In order to implement the basic functions of the message queue mentioned above:

1) message transmission

2) Stora

3) consumption

It needs to involve the following three aspects of design:

1) Communication protocol

2) Storage selection

3) maintenance of consumption relationship

Communication protocol

Message Message: since it is the carrier of information, the message sender needs to know how to construct the message, and the message receiver needs to know how to parse the message. They need to describe the message according to a unified format, which is called message protocol.

Traditional communication protocol standards include XMPP and AMQP protocols. Now more message queues use their own designed and implemented communication protocols from the point of view of performance.

1.JMS

JMS (Java MessageService) actually means JMS API. JMS is an early message standard put forward by Sun, which aims to provide unified message operation for java applications, including creating messages, sending messages, receiving messages and so on.

JMS typically consists of the following roles:

JMS provides two message models:

1) Point-to-point

2) and publish-subscribe (publish and subscribe) model.

When using the peer-to-peer model, messages are sent to a queue whose messages can only be consumed by one consumer.

When using the publish-subscribe model, messages can be consumed by multiple consumers.

In the publish and subscribe model, producers and consumers are completely independent and do not need to perceive the existence of each other.

2.AMQP

AMQP is Advanced Message Queuing Protocol, the advanced message queuing protocol.

AMQP is not a concrete message queue implementation, but a standardized message middleware protocol.

The goal is to allow applications of different languages and systems to communicate with each other, and to provide a simple and unified model and programming interface. At present, the mainstream ActiveMQ and RabbitMQ support AMQP protocol.

AMQP is a protocol, or more accurately, a binary wire-level protocol (link protocol). This is the essential difference between AMQP and JMS. AMQP does not qualify from the API layer, but directly defines the data format for network exchange.

Comparison between JMS and AMQP

JMS: only allow communication between messaging platforms based on JAVA

AMQP: AMQP allows multiple technologies to communicate with protocols simultaneously

Communication protocol of 3.Kafka

Kafka's Producer, Broker and Consumer use a set of self-designed protocols based on TCP layer. This set of protocols for Kafka is completely tailored to Kafka's own business needs.

Storage selection

For distributed systems, there are several storage options

1. Memory

two。 Local file system

3. Distributed file system

4.nosql

5.DB

Memory is obviously the fastest in terms of speed, and memory is a better choice for scenarios that allow message loss and do not require high message accumulation capabilities (such as logs).

DB is the simplest solution to achieve reliable storage, and it is suitable for scenarios with high reliability requirements and ultimate consistency (such as transaction messages). Hbase is also a good choice for scenarios where 100% data integrity is not required, performance and message accumulation are required.

In theory, in terms of speed, file system > distributed KV (persistence) > distributed file system > database, but reliability is quite the opposite.

You still need to make the most reasonable choice based on the supported business scenarios. If your message queue is used to support payments / transactions and other requirements for reliability is very high, but the requirements for performance and volume are not so high, and you do not have the time and energy to specialize in file storage system research, DB is the best choice.

Hbase is also a good choice for scenarios where 100% data integrity is not required, and where performance and message accumulation are required. Typical scenarios such as kafka messages landing can use hadoop.

Processing of consumption relationship

Now our message queue initially has the ability to dump messages.

The next important thing is to parse the send-receive relationship and deliver the message correctly.

Message queues on the market define a bunch of confusing nouns, such as Topic/Queue,Kafka in the JMS specification, Exchange in Topic/Partition/ConsumerGroup,RabbitMQ, and so on.

Apart from the phenomenon, there is nothing more than the difference between unicast and broadcasting.

The so-called unicast is point-to-point, while broadcasting is point-to-many.

In order to realize the broadcast function, we must maintain the consumption relationship. Usually, the message queue itself does not maintain the consumption subscription relationship. We can use mature systems such as zookeeper to maintain the consumption relationship and issue notifications when the consumption relationship changes.

Message queuing needs to support advanced features

Order of messages

Delivery reliability guarantee

Message persistence

Support for different message models

Multi-instance cluster function

Transaction characteristics, etc.

In addition to the above basic functions of message queuing, message queuing also needs to support transactions, message retries and other functions in some special scenarios.

At this point, I believe you have a deeper understanding of "how to design a MQ message queue from 0 to 1". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.