Shulou(Shulou.com)06/01 Report--

This article mainly introduces "how to select RocketMQ message middleware". In daily operation, I believe many people have doubts about how to select RocketMQ message middleware. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts about "how to select RocketMQ message middleware". Next, please follow the editor to study!

RocketMQ introduction

Apache RocketMQ is a distributed message middleware with low latency, high concurrency, high availability and high reliability. Message queue RocketMQ can provide distributed application systems with the ability of asynchronous decoupling and peak-cutting and valley-filling. At the same time, it also has the characteristics of massive message accumulation, high throughput and reliable retry needed by Internet applications.

RocketMQ concept

Topic: message topic, used to classify a class of messages, such as an order topic, that is, all order-related messages can be carried by this topic, and the producer sends messages to this topic.

Producer: the role responsible for producing messages and sending messages to Topic.

Consumer: the role responsible for receiving and consuming messages from Topic.

Message: the content sent by the producer to the Topic will be consumed by consumers.

Message attributes: producers can customize some business-related attributes for messages when sending them, such as Message Key and Tag.

Group: a class of producers or consumers who usually produce or consume the same type of messages and have the same logic for publishing or subscribing to messages.

Why use RocketMQ? Asynchronous decoupling

With the popularity of microservice architecture, it is very important to sort out the relationship between services. Asynchronous decoupling can not only reduce the coupling between services, but also improve the throughput of services.

There are many business scenarios that use asynchronous decoupling, because the business in each industry will be different. I believe everyone can understand it with some more general-purpose business.

For example, in the case of the order issuing business scenario in the e-commerce industry, for the simplest order issuing process, the order issuing process is as follows:

Lock inventory

Create an order

User payment

Deduction of inventory

Send a purchase SMS notification to the user

Add points to the user

Notify the merchant to deliver the goods

After the following order is successful, the user makes the payment. When the payment is completed, there will be a logic called payment callback. Some business logic needs to be done in the callback. First, let's take a look at the time it takes to synchronize, as shown in the following figure:

The above order issuing process from 3 to 5 can be handled by an asynchronous process, and for the user, he does not need to pay attention to the following process after the payment is completed. Just deal with it slowly in the background, so that you can simplify three steps and improve the processing time of callbacks.

Cut peak and fill valley

Peak cutting and valley filling refers to the use of RocketMQ to resist instantaneous large flow, protect the stability of the system and enhance the user experience under the impact of large flow.

In the e-commerce industry, the most common traffic shock is flash sale activity, using RocketMQ to achieve a complete second kill service is still a lot of work that needs to be done, outside the scope of this article, later there is an opportunity to talk to you alone. What I want to tell you is that scenarios like this can use RocketMQ to withstand high concurrency, as long as the business scenario supports asynchronous processing.

Final consistency of distributed transactions

As we all know, there are some schemes for distributed transactions, such as 2PCMagee TCC, final consistency and so on. Among them, it is common to use message queue to make the final consistency scheme.

In the business scenario of e-commerce, the core business related to transactions must ensure the consistency of data. By introducing the distributed transaction of RocketMQ version of message queuing, we can not only realize the decoupling between systems, but also ensure the final data consistency.

Data distribution

Data distribution refers to the requirement that the original data can be distributed to multiple systems that need to use the data to achieve data heterogeneity. The most common ones are distributing data to ES and providing search, caching and other services for business in Redis.

In addition to manual data distribution through the message mechanism, you can also subscribe to Mysql's binlog for distribution. In this scenario, you need to use RocketMQ's sequential messages to ensure data consistency.

RocketMQ architecture

The picture is from the official document of Ali Yun.

Name Server: an almost stateless node that can be deployed in a cluster, providing naming services, updates, and discovery Broker services in the message queuing RocketMQ version. It's a registration center.

Broker: message transfer role, responsible for storing and forwarding messages. Divided into Master Broker and Slave Broker, a Master Broker can correspond to multiple Slave Broker, but a Slave Broker can only correspond to one Master Broker. After Broker starts, it needs to register itself with Name Server, and then periodically reports Topic routing information to Name Server every 30s.

Producer: establish a long link (Keep-alive) with one of the nodes in the Name Server cluster (random), regularly read Topic routing information from the Name Server, establish a long link to the Master Broker providing Topic services, and regularly send a heartbeat to the Master Broker.

Consumers: establish a long connection (randomly) with one of the nodes in the Name Server cluster, regularly pull Topic routing information from Name Server, establish a long connection to Master Broker and Slave Broker providing Topic services, and regularly send heartbeats to Master Broker and Slave Broker. Consumer can subscribe to messages from either Master Broker or Slave Broker, and the subscription rules are determined by the Broker configuration.

RocketMQ message type

RocketMQ supports a wide range of message types to meet the business needs of multiple scenarios. Different messages have different application scenarios. Here are four commonly used message types.

General message

A normal message is a message that has no features in RocketMQ. When there is no special business scenario, it is enough to use ordinary messages. If there are special scenarios, you can use special message types, such as order, transactions, and so on.

Synchronous transmission

Synchronous sending: the sender sends a message and gets the result returned by the server synchronously.

Asynchronous transmission

Asynchronous sending: the sender sends a message, and instead of waiting for the server to return the result, the sender can send the next message. The sender can receive the server response through the callback interface and process the response result.

One-way transmission

One-way sending: the sender of the message is only responsible for sending the message, and it doesn't care after it is sent out. In this way, the sending speed is very fast and there is a risk of losing the message.

Sequential message

Sequential messages means that producers publish messages in a certain order; consumers subscribe to messages in a given order, that is, the first published messages must be received by consumers first.

For example, in the scenario of data distribution, if we subscribe to Mysql's binlog for data heterogeneity. If there is no order of messages, the problem of data disorder will occur.

For example, add a piece of id=1 data and delete it immediately. This produces two messages. The normal consumption order is to first add and then delete, at this time the data is not available. If the messages are out of order, the deleted ones are consumed first, and then the new ones are consumed, and the data is still there and has not been deleted, which will lead to inconsistency.

Timing message

Timing message means that the message has the function of sending at a fixed time, and when the message is sent to the server, it will not be delivered to the consumer immediately. Instead, it will not be delivered to consumers for consumption until the time specified by the message.

A delay message is a timing message, which is sent at a certain point in time, for example, at 12:00:00 on 2020-11-11.

A delayed message is usually sent on the basis of the current sending time. For example, if the current time is 2020-09-10 12:00:00 and the delay is 10 minutes, the message will be delivered to the consumer at 12:10:00 on 2020-09-10.

Timing messages can be used in scenarios such as automatic cancellation of order timeouts and unpaid payments.

Transaction message

RocketMQ provides distributed transaction functions similar to X/Open XA, and the final consistency of distributed transactions can be achieved through RocketMQ transaction messages.

Interaction process:

The picture is from the official document of Ali Yun.

The sender first sends a semi-transactional message to the RocketMQ server.

After receiving the message and persisting the message successfully, the RocketMQ server returns to the sender Ack confirmation that the message has been sent successfully. In this case, the message is a semi-transactional message and will not be delivered to the consumer.

After receiving the Ack of the semi-transactional message, the sender begins to execute the local transaction logic.

The sender submits a second acknowledgement to the server according to the execution result of the local transaction. If the local transaction is executed, the Commit of the message is performed. If the execution fails, the Rollback of the message is performed. When the server receives the Commit status, the semi-transactional message is marked as deliverable, and the consumer will eventually receive the message. If the server receives the Rollback status, the semi-transactional message will be deleted, and the consumer will not receive the message.

If an unexpected situation occurs, the second confirmation of the message is not carried out in step 4, and the server will initiate a message check on the message after waiting for a fixed time.

After receiving the message, the sender needs to check the final result of the local transaction execution of the corresponding message. The sender submits the second confirmation again according to the final status of the local transaction, and the server still operates the semi-transaction message according to step 4.

Best practice message retry

After the consumer fails to consume the message, the RocketMQ server will re-deliver the message and know that the consumer has successfully consumed the message. Of course, there is a limit on the number of retries. Default is 16 times.

Message retry ensures that the message will not be lost to a certain extent, and the final consumption can be achieved through retry. It should be noted that when consumers spend, they must wait for the success of the local business before ACK (consumption confirmation), otherwise the consumption will fail, but the message will not be delivered repeatedly because it is already ACK.

If you use the method of asynchronous consumption, you need to transfer from asynchronous to synchronous, and wait until the asynchronous operation is completed before ACK.

Finally, you need to do a good job of monitoring. If you retry 4 and 5 times or fail, basically you will fail to retry later. At this time, you need to let the developer know that what should be handled manually will be manually involved. Or directly monitor the dead letter queue.

Message filtering

Message topic is generally used for the unified classification of a class of messages. For example, the subject of the order, but the message under the order will be divided into many kinds. Such as creating orders, canceling orders, and so on.

Different types of messages have different business processing, we can uniformly define the message format, and then use a field to distinguish the message type to do different business logic. The downside is that all messages are pushed to the consumer and cannot be consumed on demand.

Tag can be specified for messages in RocketMQ, and message types can be distinguished by tag. Consumers can filter messages on the RocketMQ server based on Tag to ensure that consumers end up consuming only the types of messages they care about.

I have come across a way in which tag is not used correctly, there is only one instance of MQ, and tag is used to distinguish between environments. All the messages are in one topic, the test environment consumes the tag of the test environment, and the online consumption tag.

The problem with this approach is that messages are not quarantined and messages online and offline are all together. The other is that tag is fixed as the distinction of environment and cannot be used in message type scenarios. As a result, only multiple topic can be built to host multiple business message types.

Consumption pattern

There are two RocketMQ consumption patterns, cluster consumption and broadcast consumption.

Cluster consumption:

Consumers deploy multiple instances, which we call a cluster, and cluster consumption is only consumed by one of the instances.

It is suitable for most business scenarios. In most scenarios, our messages can only be consumed once, and only one consumer can consume them. For example, in the payment callback scenario, if a message is consumed by multiple instances at the same time, then the order status will be modified and the inventory will be deducted at the same time.

Broadcast consumption:

Broadcast consumption causes each instance in the cluster to be consumed once.

For example, we use a local cache, and when the data changes, we need to refresh the local cache of each node, so each node needs to receive a message.

Consumption idempotent

Idempotent problems can be encountered in both API request scenarios and message consumption scenarios. It must be guaranteed that a message cannot be consumed many times, because we cannot guarantee that the message sender will not send it many times, nor can we guarantee that the message will not be delivered repeatedly.

The Exactly-Once delivery semantics of RocketMQ is used to solve idempotent problems. Exactly-Once means that the message sent to the messaging system can only be processed by the consumer and processed only once. Even if the producer retries sending the message and causes a message to be delivered repeatedly, the message is consumed only once on the consumer side.

The best idempotent processing method still requires a unique business identity. Although each message has a MessageId, it is not recommended to use MessageId to make idempotent judgment. When sending a message, you can set a MessageKey for each message, and this MessageKey can be used as the unique identity of the business.

Local transaction message encapsulation

The transaction message is introduced above, and the transaction message of RocketMQ adopts a two-phase commit method. And the mechanism of message contrast is combined to ensure the final consistency.

In terms of usage, it is a bit annoying that every business scenario has to implement a contrasting logic.

Another frequently used approach, local transaction messages, is described below. The scheme of local message table was originally proposed by ebay. Local transaction messages need to create a message table in the corresponding database of the service. When sending a message, the message is not really sent to MQ, but a message data is inserted into the message table.

The inserted action is the same transaction as the local business logic. If the local transaction executes successfully, the message will be successfully listed and sent to MQ. The local transaction fails and the message data will be rolled back.

Then you need a special program to pull unsent messages from the message table and deliver them to MQ. If the delivery fails, you can try again until successful or manual intervention.

The message is written to the message table and then sent to MQ all the time. This step is fine. If the Broker goes down after MQ receives the message and the message is still in PageCache, the message will be lost at this time. Of course, you can also use synchronous disk brushing and other methods to avoid loss. If we swipe the disk asynchronously, is there any way to ensure that the message is not lost?

As we mentioned earlier, RocketMQ transaction messages will have a mechanism to check back, message table way, also need to have a mechanism to ensure that the message is consumed, otherwise you need to keep retrying to send the message until the message is consumed.

There needs to be a field in the message table to identify the current status of the message, such as unsent, sent, and consumed. When the message is still unsent, it is sent to MQ, and if it is sent successfully, the status is that it has been sent. But after a few minutes, the status has been sent, and it's time to do something.

In this scenario, it is possible that consumers can not keep up with the speed of production, and messages are piled up, resulting in messages that have not been consumed. Another possibility is whether the news is lost.

You can get the corresponding message stacking data to determine whether the message is stacked, and if not, resend the message to MQ until the message is consumed.

The problem is that the news has been consumed, how do I know?

Like the cloud service I use, there is a corresponding Open API that can directly query the message track. Open source should also be available, without careful study, and the commercial version should be similar.

According to the message trajectory, you can know whether the message has been consumed or not, and this ends the process. If the message is sent to MQ, it will be retried if it is not consumed for a long time. Even if it finally enters the dead-letter queue, it can be manually intervened through the monitoring of the dead-letter queue, which must be the final consistency.

Compared with the built-in transaction messages, the local message table does not need to implement the lookback logic, but it is also troublesome to add message tables and to support a variety of sending, checking and other logic. Especially when the message volume is large, how to quickly send out the messages in the message table also needs to do a lot of processing, simple look-up table polling is not suitable in the case of a large number of messages. Both methods can be used to achieve the goal we want.

At this point, the study on "how to choose RocketMQ message middleware" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.