Shulou(Shulou.com)06/02 Report--

This article will explain in detail how to carry out the summary and analysis of Java message queue. The content of the article is of high quality, so the editor will share it with you for reference. I hope you will have some understanding of the relevant knowledge after reading this article.

Overview of message queuing

Message queuing middleware is an important component in distributed system, which mainly solves the problems of application decoupling, asynchronous messaging, traffic sharpening and so on, and achieves high performance, high availability, scalability and final consistency architecture. At present, the most commonly used message queue is ActiveMQ,RabbitMQ,ZeroMQ,Kafka,MetaMQ,RocketMQ.

Second, message queuing application scenario

The following describes the commonly used scenarios of message queuing in practical applications. There are four scenarios: asynchronous processing, application decoupling, traffic sharpening and message communication.

2.1 Asynchronous processing

Scenario description: after registration, users need to send registration email and registration SMS. There are two traditional practices. Serial mode; 2. Parallel mode

Serial mode: after the registration information is successfully written into the database, send the registration email, and then send the registration text message. When all the above three tasks are completed, return to the client.

B, parallel mode: after the registration information is successfully written into the database, send the registration email and send the registration text message at the same time. After the above three tasks are completed, return to the client. The difference with serial is that parallel approach can improve processing time.

Assuming that each of the three business nodes uses 50 milliseconds, regardless of other overhead such as the network, the serial time is 150 milliseconds, and the parallel time may be 100 milliseconds.

Because there is a certain number of requests processed by CPU per unit time, it is assumed that the throughput of CPU1 is 100 per second. Then the number of requests that can be processed by CPU in serial mode in 1 second is 7 (1000,150). The number of requests processed in parallel is 10 times (1000 Universe 100)

Summary: as described in the above case, the traditional system performance (concurrency, throughput, response time) will have a bottleneck. How to solve this problem?

The introduction of message queuing will not be a necessary business logic for asynchronous processing. The modified structure is as follows:

According to the above convention, the response time of the user is equal to the time that the registration information is written to the database, that is, 50 milliseconds. Register the mail, send a short message to write to the message queue, and return directly, so the speed of writing to the message queue is very fast and can be ignored, so the user's response time may be 50 milliseconds. Therefore, after the architecture change, the throughput of the system is increased to 20 QPS per second. It is three times higher than serial and two times higher than parallel.

2.2 Application decoupling

Scenario description: after the user places an order, the order system needs to notify the inventory system. Traditionally, the order system calls the interface of the inventory system. As shown below:

Disadvantages of the traditional model: if the inventory system is not accessible, the order reduction will fail, resulting in order failure, and the order system is coupled with the inventory system.

How to solve the above problems? The solution after introducing the application message queue, as shown below:

Order system: after the user places an order, the order system completes the persistence processing, writes the message to the message queue, and returns the user's order to be issued successfully.

Inventory system: subscribe to the order message and use the pull / push method to obtain the order information. The inventory system carries out inventory operation according to the order information.

If: the inventory system does not work properly when placing an order. It also does not affect the normal issuance of the order, because after the order is issued, the order system writes to the message queue and no longer cares about other subsequent operations. Realize the application decoupling of order system and inventory system

2.3 flow peaking

Traffic sharpening is also a common scenario in message queues, which is widely used in second kill or group robbery activities.

Application scenario: flash sale activity, usually because the traffic is too large, the traffic will surge and the application will fail. In order to solve this problem, it is generally necessary to join the message queue at the front end of the application.

A. The number of people who can control the activity

B, it can relieve the application of high flow collapse in a short time.

After receiving the user's request, the server first writes to the message queue. If the length of the message queue exceeds the number of messages, the user request will be discarded directly or jump to the error page.

The second kill service does subsequent processing according to the request information in the message queue.

2.4 days of log processing

Log processing refers to the use of message queues in log processing, such as the application of Kafka, to solve the problem of mass log transmission. The architecture is simplified as follows:

The log collection client is responsible for log data collection and writes regularly to the Kafka queue.

Kafka message queue, responsible for receiving, storing and forwarding log data.

Log processing applications: subscribe to and consume log data in kafka queues.

2.5 message communication

Message communication means that message queues generally have built-in efficient communication mechanisms, so they can also be used in pure message communication. Such as implementing peer-to-peer message queues, or chat rooms, etc.

Point-to-point communication:

Client An and client B use the same queue for message communication.

Chat room newsletter:

Client A, client B and client N subscribe to the same topic to publish and receive messages. Achieve a chat room-like effect.

The above are actually two message modes of message queuing, peer-to-peer or publish-subscribe mode. The model is a schematic diagram for reference.

Third, message middleware example

3.1 E-commerce system

Message queuing uses highly available and persistent message middleware. Like Active MQ,Rabbit MQ,Rocket Mq.

After the application completes the trunk logic processing, it is written to the message queue. Whether the message is sent successfully or not can turn on the confirmation mode of the message. (after message queuing returns the status of successful message reception, the application returns, thus ensuring the integrity of the message.)

Expand the process (SMS, delivery processing) to subscribe to queue messages. Use push or pull to get the message and process it.

While the message decouples the application, it brings the problem of data consistency, which can be solved in the final consistency way. For example, the master data is written into the database, and the extended application realizes the subsequent processing based on the message queue according to the message queue and the database method.

3.2 Chronicle Collection system

It is divided into four parts: Zookeeper registry, log collection client, Kafka cluster and Storm cluster (OtherApp).

Zookeeper registry, propose load balancing and address lookup.

The log collection client collects logs from the application system and pushes the data to the kafka queue

Kafka cluster: receiving, routing, storing, forwarding and other message processing

Storm cluster: at the same level as OtherApp, the data in the queue is consumed by pull.

MQ selection comparison document

Comprehensive selection RabbitMq

Kafka is an open source MQ system of linkedin. Its main feature is to deal with message consumption based on Pull mode and pursue high throughput. Its initial purpose is for log collection and transmission. It supports replication and does not support transactions. It is suitable for data collection business of Internet services that generate a large amount of data.

RabbitMQ is an open source message queuing system developed in Erlang language and implemented based on AMQP protocol. The main features of AMQP are message-oriented, queue-oriented, routing (including peer-to-peer and publish / subscribe), reliability and security. AMQP protocol is more used in enterprise systems, requiring high data consistency, stability and reliability, followed by performance and throughput requirements.

RocketMQ is Ali open source message middleware, it is pure Java development, with high throughput, high availability, suitable for large-scale distributed system applications. The idea of RocketMQ originated from Kafka, but it is not a Copy of Kafka. It optimizes the reliable transmission and transactionality of messages. At present, it is widely used in trading, recharging, flow computing, message push, log streaming, binglog distribution and other scenarios.

ZeroMQ is just a Pattern library for network programming, which models and componentizes the common forms of network requests (packet management, link management, publish and subscribe, etc.), in short, above socket and below MQ. For MQ, network transport is only a part of it, and more needs to deal with message storage, routing, Broker service discovery and lookup, transactions, consumption patterns (ack, reinvestment, etc.), cluster services, and so on.

RabbitMQ/Kafka/ZeroMQ can provide message queuing service, but there is a big difference.

In service-oriented architecture, it is a good idea to use producer-consumer pattern to communicate asynchronously between services through message broker (such as RabbitMQ / Kafka, etc.).

Because inter-service dependencies have changed from strong coupling to loose coupling. Message agents provide a persistence mechanism to save messages without loss in the event of heavy consumer load or disconnection. That is to say, producers and consumers do not need to be online at the same time, which is difficult to do in the traditional request-response model, and a middleware is needed to do this. Secondly, the message broker can make a simple routing strategy according to the message itself, and consumers can do load balancing, business separation and so on.

There is also a disadvantage, which is that additional message broker clusters are needed (but the advantages outweigh the disadvantages).

Unlike RabbitMQ/Kafka, ZeroMQ is an asynchronous message library that provides a mechanism similar to a message broker based on sockets. If you use ZeroMQ, you need to modify your business code, which is not conducive to service decoupling.

RabbitMQ supports AMQP (binary), STOMP (text), MQTT (binary), HTTP (packaging other protocols) and other protocols. Kafka uses its own protocol.

Both Kafka's own services and consumers rely on Zookeeper.

RabbitMQ performance degrades when there is a large number of messages piled up, but Kafka does not. After all, AMQP was not designed to persist massive messages, while Kafka was originally designed to handle massive logs.

Generally speaking, RabbitMQ and Kafka are excellent distributed message broker services. As long as they are reasonably deployed and not done, they can basically meet any needs under production conditions.

This is the end of the summary and analysis on how to carry out the Java message queue. I hope the above content can be helpful to you and learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.