Shulou(Shulou.com)06/01 Report--

This article mainly talks about "what are the web message queues?". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn what web message queues have.

Message queue

Message queuing middleware is an important component in distributed systems, which mainly solves the problems of application coupling, asynchronous messaging, traffic sharpening and so on. Message queues with high performance, high availability, scalability and final consistency architecture are widely used in ActiveMQ,RabbitMQ,ZeroMQ,Kafka,MetaMQ,RocketMQ and so on.

Application scenario

Message queuing is a common usage scenario in practical applications. There are four scenarios: asynchronous processing, application decoupling, traffic sharpening and message communication.

Asynchronous processing

Scenario description: after registration, users need to send registration email and registration SMS.

There are two traditional practices. Serial mode; 2. Parallel mode

Serial mode: after the registration information is successfully written into the database, send the registration email, and then send the registration text message. When all the above three tasks are completed, return to the client.

Parallel mode: after the registration information is successfully written into the database, the registration email is sent and the registration SMS is sent at the same time. After the above three tasks are completed, return to the client. The difference with serial is that parallel approach can improve processing time.

Assuming that each of the three business nodes uses 50 milliseconds, regardless of other overhead such as the network, the serial time is 150 milliseconds, and the parallel time may be 100 milliseconds.

Because there is a certain number of requests processed by CPU per unit time, it is assumed that the throughput of CPU1 is 100 per second. Then the number of requests that can be processed by CPU in serial mode in 1 second is 7 (1000,150). The number of requests processed in parallel is 10 times (1000 Universe 100).

After the introduction of message queuing, it will not be a necessary business logic for asynchronous processing.

The response time of the user is equal to the time that the registration information is written to the database, that is, 50 milliseconds. Register the mail, send a short message to write to the message queue, and return directly, so the speed of writing to the message queue is very fast and can be ignored, so the user's response time may be 50 milliseconds. Therefore, after the architecture change, the throughput of the system is increased to 20 QPS per second. It is three times higher than serial and two times higher than parallel.

Application decoupling

After the user places an order, the order system needs to notify the inventory system. Traditionally, the order system calls the interface of the inventory system.

Avoid the interface and join the message queue.

Order system: after the user places the order, the order system completes the persistence processing, writes the message into the message queue, and returns the user order to issue the order successfully.

Inventory system: subscribe to the order message and use the pull / push method to obtain the order information. The inventory system carries out the inventory operation according to the order information.

If: the inventory system does not work properly when placing an order. It also does not affect the normal issuance of the order, because after the order is issued, the order system writes to the message queue and no longer cares about other subsequent operations. Realize the application decoupling of order system and inventory system.

Flow cutting edge

Traffic sharpening is also a common scenario in message queues, which is widely used in second kill or group robbery activities.

Flash sale activity, usually because the traffic is too large, resulting in a sudden increase in traffic, the application is down. In order to solve this problem, it is generally necessary to join the message queue at the front end of the application.

The number of people who can control the activity

It can relieve the application of high flow collapse in a short time.

After receiving the user's request, the server first writes to the message queue. If the length of the message queue exceeds the maximum, the user request is discarded directly or jumps to the error page.

The second kill service does subsequent processing according to the request information in the message queue.

Message communication

Log processing refers to the use of message queues in log processing, such as the application of Kafka, to solve the problem of mass log transmission.

The log collection client is responsible for log data collection and writes regularly to the Kafka queue.

Kafka message queue, responsible for receiving, storing and forwarding log data.

Log processing applications: subscribe to and consume log data in kafka queues.

Queue comparison under large amount of data

Several representative MQ (rabbitMQ,kafka,ActiveMQ) to make a simple comparison.

The specific comparisons are as follows:

TPS comparison:

Kafka is the highest, RabbitMq is the second, and ActiveMq is the worst.

Throughput comparison:

Kafka has high throughput, internal use of message batch processing, zero-copy mechanism, data storage and acquisition is a local disk sequential batch operation, with O (1) complexity, message processing efficiency is very high. RabbitMQ is slightly inferior to kafka in terms of throughput, and their starting point is different. RabbitMQ supports reliable delivery of messages, supports transactions, and does not support batch operations; storage-based reliability requires that storage can be stored in memory or hard disk.

In terms of architectural model:

RabbitMQ follows the AMQP protocol, and the broker of RabbitMQ is composed of Exchange,Binding,queue, where exchange and binding constitute the routing key of the message; the client Producer communicates with server by connecting channel, and Consumer obtains messages from queue for consumption (persistent connection, messages in queue are pushed to the server, and consumer loop reads data from the input stream). RabbitMQ takes broker as the center; there is a message confirmation mechanism.

Kafka follows the general MQ structure, producer,broker,consumer, with consumer as the center, on the client consumer where the consumption information of the message is saved, and the consumer batches the pull data from the broker according to the point of consumption; there is no message confirmation mechanism.

In terms of usability

RabbitMQ supports miror's queue, the main queue fails, and miror queue takes over. Kafka's broker supports active / standby mode.

ActiveMq also supports active / standby mode.

In the aspect of cluster load balancing

Kafka uses zookeeper to manage broker and consumer in the cluster, and can register topic to zookeeper; through the coordination mechanism of zookeeper, producer saves the broker information of the corresponding topic, which can be randomly or polled and sent to broker; and producer can specify shards based on semantics, and messages can be sent to a shard of broker.

The load balancing of rabbitMQ needs to be supported by a separate loadbalancer.

Comprehensive comparison

ActiveMQ: an open source project with a long history, has been applied in many products, implemented the JMS1.1 specification, can be easily integrated with spring-jms, implemented a variety of protocols, is not lightweight enough (more source code than RocketMQ), supports persistence to the database, and does not support a large number of queues.

RabbitMq: it is more mature than kafka and supports AMQP transaction processing. RabbitMq surpasses kafka in reliability and ActiveMQ in performance.

Kafka: Kafka is designed to deal with logs, does not support AMQP transactions, can be seen as a logging system, targeted, so it does not have a mature MQ should have the characteristics of Kafka performance (throughput, tps) is stronger than RabbitMq, if used to do a large amount of data fast processing is better than RabbitMq.

Finally, a comprehensive comparison is made:

At this point, I believe you have a deeper understanding of "what web message queues have". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.