Shulou(Shulou.com)06/03 Report--

In this issue, the editor will bring you about what is a message queue. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

Many people tend to "over-design" when doing architectural design, and simple problems become more complicated, so they quote a bunch of middleware. I think the main reasons are the following two points:

To structure in order to show (learn) skills.

We often say that technology is for business service, not technology for technology's sake, and it is not desirable to introduce a bunch of complex architectures in order to show technology.

The consideration of the problem is not comprehensive, or the breadth is not enough, do not know how to simplify

Take Chestnut, for example, suppose there is a highly concurrent user platform that needs to handle registration (write) and login query (read) functions, and does master-slave synchronization in the database layer.

In order to solve the problem of master-slave synchronization delay, someone introduced a Redis to write Redis when writing master library, and then read Redis directly when reading, in order to avoid the problem of master-slave delay synchronization. This is a typical problem that is not considered comprehensively. Although this can solve the problem of master-slave delay, it will lead to the problem of double-write consistency transactions. Then it is better to directly change the master-slave synchronization mode to strong synchronous replication to ensure consistency directly from the database level.

Then you may say that changing to strong synchronous replication will not increase the response time and then affect the system throughput, then we can also do a sub-library for users and do more master-slave synchronization, can't we?

Well, it's beside the point. Didn't we talk about message queue today?

Oh, let's get down to business. Today we talk about the problem of message queue. I hope you will think about it when you introduce message queue after reading this article. Is it necessary to introduce message queue? Can the problems caused by the introduction of message queues be solved?

The role of message queues

In the development of micro services, we often introduce message middleware to decouple business and perform asynchronous operations. now let's look at the advantages and disadvantages of using message middleware.

First of all, we need to be sure that there are many benefits of using message components, of which the three core ones are decoupling, asynchronism, and peaking.

Decoupling: the client only needs to send the request to a specific channel and does not need to be aware of receiving the request instance.

Async: messages are written to the message queue, and unnecessary business logic runs asynchronously to speed up the response.

Peaking: message middleware caches messages before they are consumed, and message processors can slowly process messages from the message queue according to their own concurrency, without crushing the business in an instant.

Of course, message middleware is not a silver bullet, and the introduction of message mechanism will also have the following disadvantages:

Potential performance bottlenecks: message agents may have performance bottlenecks. Fortunately, the current mainstream message middleware supports a high degree of scale-out.

Potential single point of failure: the high availability of the message broker is critical, otherwise the overall reliability of the system will be affected. Fortunately, most message middleware are highly available.

Additional operational complexity: the messaging system is a system component that must be installed, configured and operated independently, which increases the complexity of the operation and maintenance.

These drawbacks can be solved with the help of the extension and high availability provided by the message middleware itself, but we still need to pay attention to some design problems that may be encountered in order to really use the good news middleware.

Message queue Design challenges dealing with concurrent and Sequential messages

In order to improve the ability of message processing and the throughput of applications in a production environment, consumers are generally deployed with multiple instance nodes. The challenge is to ensure that each message is processed only once and in the order in which it is sent.

For example, suppose there are three identical receiver instances that read messages from the same point-to-point channel, and the sender publishes Order Created, Order Updated, and Order Cancelled event messages in order. A simple message implementation may tell a colleague each message to a different receiver. If there is a delay due to network problems, messages may not be processed in the order in which they were sent, which leads to strange behavior, and the service instance may process Order Created messages before another server processes Order Cancelled messages.

The solution used by Kafka is to use sharded (partitioned) channels. The overall solution is divided into three parts:

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

A topic channel consists of multiple shards, and each shard behaves like a channel.

The sender specifies a sharding key in the message header such as orderId,Kafka uses the sharding key to assign the message to a specific shard.

Combine multiple instances of the receiver and treat them as the same logical receiver (consumer group). Kafka allocates each shard to a single receiver, which redistributes shards when the receiver starts and shuts down.

As shown in the figure above, each Order event message has orderId as its sharding key. Each event for a specific order is published to the same shard. And the messages in the fragment are always read by the same receiver instance, so it is guaranteed that the messages are processed sequentially.

Handle duplicate messages

Another challenge that must be addressed in introducing a message architecture is to deal with duplicate messages. Ideally, the message broker should deliver the message only once, but the cost of ensuring that the message is delivered once and only once is usually high. Instead, many messaging components promise at least one successful delivery of the message.

Under normal circumstances, the message component delivers the message only once. However, when a client, network, or message component fails, the message can be delivered multiple times. Suppose the client crashes its database before sending an acknowledgement message after processing the message, and the message component will send the unacknowledged message again to the client when the database is restarted.

There are two different ways to handle duplicate messages:

Write idempotent message handlers

Track messages and discard duplicates

Write idempotent message handlers

If the logic of the application processing messages is idempotent, then repeating messages is harmless. The idempotency of the program means that even if the application is called repeatedly with the same input parameter, it will not have any additional effect. For example, canceling an order that has been cancelled is an idempotent operation. Similarly, this must be the case for creating an existing order operation. Idempotent message handlers can be safely executed multiple times, as long as the message component maintains the same message order when delivering messages.

Unfortunately, applications are not usually idempotent. Or the message component you are using now will not retain the sort when the message is redelivered. Duplicate or unordered messages can cause errors. In this case, you need to write a message handler that tracks messages and discards duplicate messages.

Track messages and discard duplicate messages

Consider a message handler that authorizes consumer credit cards. It must perform a credit card authorization operation only once per order. This application has a different effect each time it is called. If duplicate messages cause the message handler to execute the logic multiple times, the application will behave incorrectly. Message handlers that perform such application logic must make duplicate messages idempotent by detecting and discarding them.

A simple solution is for the message receiver to use message id to track the messages he has processed and discard any duplicates. For example, store the message id of each message it consumes in a database table.

When the receiver processes the message, it records the message id of the message in the data table as part of the transaction that creates and changes the business entity. As shown in the figure above, the receiver inserts the row containing message id into the PROCESSED_ messages table. If the message is duplicated, the INSERT will fail and the receiver can choose to discard the message.

Another solution is for the message handler to record the message id in the application table instead of the specialty table. This approach is particularly useful when using NoSQL databases with a restricted transaction model, because NoSQL databases generally do not support updates to two tables as database transactions.

Handling transactional messages

The service usually needs to publish the message in a transaction that updates the database, and both the database update and message delivery must take place in the transaction, otherwise the service may update the database and crash before sending the message.

If the service does not perform two operations atomically, a similar failure may put the system in an inconsistent state.

Next, let's take a look at the two commonly used solutions to guarantee transactional messages, and finally take a look at the transactional messaging solution of modern messaging component RocketMQ.

Use database tables as message queues

If your application is using a relational database, make sure that transactions between data updates and message delivery can directly use the transactional outbox mode, Transactional Outbox.

This pattern uses database tables as temporary message queues. As shown in the figure above, the service that sends the message has an OUTBOX data table, and it also gives the OUTBOX data table INSERT a message record when performing INSERT, UPDATE, and DELETE business operations, which ensures atomicity because it is based on local ACID transactions.

The OUTBOX table acts as a temporary message queue, and then we are introducing a message relay (MessageRelay) service that reads data from the OUTBOX table and publishes messages to the message component.

The implementation of message relay can be very simple, only need to pull the latest unpublished data from the OUTBOX table periodically through the scheduled task, get the data and send the data to the message component, and finally delete the message from the OUTBOX table.

Publish events using the transaction log

Another way to guarantee transactional messages is based on the database transaction log, which is called data change capture, Change Data Capture, or CDC.

Generally, a database records a transaction log (Transaction Log) when the data changes, such as MySQL's binlog. The transaction log can be simply understood as a file queue local to the database, which mainly records database table changes in chronological order.

Here we use alibaba's open source component canal combined with MySQL to illustrate how this pattern works.

For more instructions, please refer to the official document: https://github.com/alibaba/canal

How canal works

Canal simulates the interaction protocol of MySQL slave, disguises itself as a slave node of MySQL, and sends dump protocol to MySQL master.

MySQL master receives the dump request and starts to push binary log to slave (i.e. canal)

Canal parses the binary log object (originally a byte stream), and then sends the parsed data directly to the message component.

RocketMQ transaction message solution

Apache RocketMQ has supported distributed transaction messages in version 4.3.0. RocketMQ adopts the idea of 2PC to implement commit transaction messages, while adding a compensation logic to handle two-phase timeout or failure messages, as shown in the following figure.

The realization of transaction message by RocketMQ is mainly divided into two stages: the sending and committing of normal transaction and the compensation process of transaction information.

The overall process is as follows:

Normal transaction send and commit phase

1. The producer sends one and a half messages to MQServer (semi-messages refer to messages that consumers cannot consume temporarily)

2. The server response message is written into the result, and the semi-message is sent successfully.

3. Start executing local transactions

4. Perform Commit or Rollback operations according to the execution status of the local transaction

Compensation process of transaction information

1. If the MQServer does not receive the execution status of the local transaction for a long time, it will issue an operation request to the producer to confirm the check back.

2. After receiving the confirmation request, the producer checks the execution status of the local transaction.

3. According to the results of the check, the compensation phase of Commit or Rollback operation is mainly used to solve the situation that the producer times out or fails when sending Commit or Rollback operation.

When producers use RocketMQ to send transaction messages, we will also learn from the first solution, that is, create a transaction log table, and then generate a transaction log record when executing the local transaction, so that the local transaction and the log transaction are in the same method, and the @ Transactional annotation is added at the same time to ensure that the two operational transactions are an atomic operation. In this way, if there is information about this local transaction in the transaction log table, it means that the local transaction was executed successfully and Commit is required. On the contrary, if there is no corresponding transaction log, it means that the execution is not successful and Rollback is required.

The above is what the editor shares with you what is a message queue. If you happen to have similar doubts, please refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.