Shulou(Shulou.com)06/01 Report--

This article mainly introduces the relevant knowledge of "what is the concept of MQ message queue". The editor shows you the operation process through an actual case. The operation method is simple, fast and practical. I hope this article "what is the concept of MQ message queue" can help you solve the problem.

I. message queuing usage scenario

1.1 Common usage scenarios

System decoupling

In the distributed environment, the interdependence between systems will eventually lead to confusion of the whole dependency relationship, especially in the micro-service environment, there will be interdependence, or even circular dependence, which will bring great burden to the splitting and optimization of the later system. Then we can use MQ to deal with it. Upstream system delivers data to MQ, downstream system takes MQ data for consumption, delivery and consumption can be processed synchronously, because the performance of MQ receiving data is very high and will not affect the performance of upstream system.

Asynchronous processing

If synchronization is adopted, there will be bottlenecks in system performance (concurrency, throughput, response time). How to solve this problem? Message queue is introduced to deal with unnecessary business logic asynchronously.

Asynchronous processing can also lead to the use of parallel processing posture. In our work, we have developed a simple distributed task processing component based on messages. The component is simply divided into three parts: slicing, loading and execution.

Each stage is used as a consumer, and then after processing, the message is sent as a producer. Message consumption is stateless and can be expanded indefinitely on demand.

Flow peaking

Due to the use of messages, our link becomes a process for producers to send messages, message middleware to store messages, and finally consumers to pull messages from message middleware. The storage capacity of message middleware can effectively help consumers to buffer. Just imagine, consumers can consume happily under normal flow. When the instantaneous peak traffic comes, consumers' spending power can not keep up, and it just blocks in the message middleware. After the peak, consumers can quickly consume the blocked messages.

Traffic sharpening is also a common scenario in message queues, which is widely used in second kill or group robbery activities.

Data distribution

Most open source MQ middleware basically supports one-to-many or broadcast mode, and you can choose the objects to be distributed according to the rules. In this way, a piece of data upstream, in many downstream systems, can choose whether to receive the data or not according to the rules, so it is very scalable.

1.2 prerequisites for message use

The above four are the most common scenarios for MQ middleware, but let's think about what problems will be caused by the introduction of MQ middleware. That's real-time. Therefore, the prerequisite for the use of MQ middleware is that it can tolerate delay, and it is more appropriate to require only final consistency.

II. Concepts related to messages

Characteristics of MQ

fifo

Can not be first-in, first-out, can not be said to be a queue. The order of message queues is basically determined when they join the queue, and generally there is no need for human intervention. And, most importantly, data is that there is only one piece of data in use. This is why MQ is used in many scenarios.

Publish and subscribe

Publish and subscribe is a very efficient way of processing, if there is no blocking, it can basically be regarded as a synchronous operation. This processing method can effectively improve the server utilization, and this kind of application scenario is very extensive.

Persistence

Persistence ensures that the use of MQ is not just an auxiliary tool for some scenarios, but rather allows MQ to store core data like a database.

Distributed system

In the current scenario of large traffic and big data, server software that only supports single applications can not be used. Only by supporting distributed deployment can it be widely used. Moreover, the positioning of MQ is a high-performance middleware.

In the JMS standard, there are two message models, P2P (Point toPoint) and Publish/Sub (Pub/Sub).

P2P

Peer-to-peer, one hair, one consumption. The roles involved are publisher (Publisher), consumer (Consumer), message queue (Queue)

Characteristics

A message can only be consumed by one consumer and will be removed from the queue after consumption.

Publishers have nothing to do with consumers, and the behavior of publishers in sending messages will not change with consumers.

If the consumer consumes the completed message, it needs to Ack to the queue. If the message queue discovers that the message is consumed successfully, the message will be removed.

Pub/Sub

Publish and subscribe model, one publication, multi-party subscription. The roles involved are publisher (Publisher), topic (Topic), and subscriber (Subscriber).

Characteristics

There can be multiple consumers per message

Subscribers to a topic (Topic) must create a subscriber before they can consume the publisher's messages

In order to consume messages, subscribers must keep running

Common problems and solutions

Message blocking

1. Message congestion usually results in a surge in traffic, which exceeds the consumption capacity of consumers.

2, or consumers have logic problems, resulting in constant retries or long waits.

The first one can be solved by expanding capacity.

The second method can only repair the problem urgently and release it online, which will cause a large backlog of messages in the process of blocking. In this case, temporary expansion can also be considered.

Repeated consumption

Repeated consumption usually occurs at the consumer end, for example, when the consumer has finished processing, there is a problem when preparing for ack. After the application is restarted, the message middleware thinks that the message has not been processed and pushed to the consumer, or it is repeated when the consumer pulls it.

The general practice is to do idempotency on the consumer side.

Message loss

Message loss is generally divided into producer failure, message middleware loss and consumption loss.

Producer loss: may be due to network problems or message intermediate processing failure, message omission.

Loss in the middle of the message: general middleware can set a discard policy, and most MQ middleware products can guarantee that data is not lost, which is basically ignored.

Consumption loss: some message middleware supports automatic ack. When consumers consume a message, the message middleware automatically ack regardless of whether the consumption is successful or not. At this time, it is more appropriate to choose consumer active ack.

Message orderliness

Message ordering is generally guaranteed by MQ middleware, most MQ middleware can only achieve local order, such as Kafka, can only guarantee the order of a single partition queue. Some will also achieve overall order, but the cost is relatively high. The company I currently serve now supports the overall order.

This is the end of the introduction to "what is the concept of MQ message queue". Thank you for reading. If you want to know more about the industry, you can follow the industry information channel. The editor will update different knowledge points for you every day.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.