Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "how to design message queues for java development microservice architecture". In the operation process of actual cases, many people will encounter such difficulties. Next, let Xiaobian lead you to learn how to deal with these situations! I hope you can read carefully and learn something!

The Role of Message Queue

In microservices development we often introduce message middleware to achieve business decoupling and perform asynchronous operations. Now let's take a look at the benefits and disadvantages of using message middleware.

First of all, there must be many benefits to using message components, of which the three most core are: decoupling, asynchronous, and clipping.

Decoupling: The client only needs to say that the request is sent to a specific channel, and does not need to be aware of the situation of receiving the request instance.

Asynchronous: Write messages to message queues, and unnecessary business logic runs asynchronously, speeding up response times.

Peak clipping: Message middleware caches messages before they are consumed. Message processors can process messages slowly from message queues according to the concurrency they process, without crushing services in an instant.

Of course, message middleware is not a silver bullet. After introducing message mechanism, there will be some disadvantages as follows:

Potential performance bottlenecks: Message brokers may have performance bottlenecks. Fortunately, the current mainstream messaging middleware supports a high degree of horizontal scalability.

Potential single point of failure: high availability of message brokers is critical, otherwise overall system reliability suffers. Fortunately, most message middleware is highly available.

Additional operational complexity: A messaging system is a system component that must be installed, configured, and operated independently, increasing the complexity of operation and maintenance.

These drawbacks can be solved with the extended and highly available capabilities provided by the messaging middleware itself, but to really use the good messaging middleware we need to pay attention to some design challenges that may be encountered.

Message Queue Design Challenges Handling Concurrent and Sequential Messages

In a production environment, consumers are typically deployed to multiple instance nodes in order to improve message processing capabilities and application throughput. The challenge then is to ensure that each message is processed only once, and in the order in which it was sent.

For example, suppose there are three identical receiver instances reading messages from the same point-to-point channel, and the sender publishes Order Created, Order Updated, and Order Cancelled in sequence. A simple message implementation might be to deliver each message to a different recipient. If delays are caused by network problems, messages may not be processed in the order in which they were sent, which can lead to strange behavior and service instances may process Order Cancelled messages before another server processes Order Created messages.

The solution Kafka uses is to use sharded (partitioned) channels. The overall solution is divided into three parts:

A topic channel consists of multiple shards, each of which behaves like a channel.

The sender specifies a fragmentation key such as orderId in the message header, and Kafka uses the fragmentation key to assign the message to a specific fragmentation.

Group multiple instances of receivers together and treat them as the same logical receiver (consumer group). Kafka assigns each fragment to a single receiver, and it reassigns fragments as the receiver starts up and shuts down.

As shown above, each Order event message has orderId as its fragmentation key. Each event for a particular order is published to the same shard. And the messages in that fragment are always read by the same receiver instance, so that they are guaranteed to be processed sequentially.

Handling duplicate messages

Another challenge that must be addressed when introducing message architectures is dealing with duplicate messages. Ideally, a message broker should deliver a message only once, but the cost of ensuring that messages are delivered once and only once is often high. Instead, many message components promise to deliver messages successfully at least once.

Under normal circumstances, a message component delivers a message only once. But when the client, network, or message component fails, it can cause messages to be delivered multiple times. Suppose that the client's database crashes before sending an acknowledgement after processing the message, and the message component sends an unacknowledged message again to the client when the database restarts.

There are two different ways to handle duplicate messages:

Writing idempotent message handlers

Track messages and discard duplicates

Writing idempotent message processors

Duplicate messages are harmless if the logic of the application's message processing is idempotent. An idempotent application is one that produces no additional effect even if it is called multiple times with the same input parameters. For example, canceling a cancelled order is an idempotent operation. Similarly, creating an existing order must be the same. An idempotent message handler can be safely executed multiple times as long as message components maintain the same message order when delivering messages.

Unfortunately, applications are not usually idempotent. Or maybe the message component you're using now doesn't preserve ordering when redelivering messages. Repeated or out-of-order messages can cause errors. In this case, you need to write message handlers that track messages and discard duplicate messages.

Track messages and discard duplicate messages

Consider a message handler that authorizes consumer credit cards. It must perform only one credit card authorization operation per order. This application has a different effect each time it is invoked. If repeated messages cause the message handler to execute the logic multiple times, the application will behave incorrectly. A message handler that implements such application logic must make it idempotent by detecting and discarding duplicate messages.

A simple solution is for the message receiver to use the message id to keep track of the messages he has processed and discard any duplicates. For example, the message id of each message it consumes is stored in a database table.

When the receiver processes the message, it records the message id in the data table as part of the transaction to create and change the business entity. As shown in the figure above, the receiver inserts the row containing the message id into the PROCESSED_MESSAGE table. If the message is a duplicate, INSERT will fail and the receiver may choose to discard the message.

Another solution is for the message handler to record the message id in the application table, rather than a dedicated table. This approach is particularly useful when using NoSQL databases with restricted transaction models, because NoSQL databases typically do not support updates to two tables as database transactions.

Processing transactional messages

Services typically need to publish messages in a transaction that updates the database, and both database updates and message delivery must occur in a transaction, otherwise the service may update the database and crash before sending the message.

If the service does not perform both operations atomically, a similar failure may leave the system in an inconsistent state.

Let's look at two common solutions for guaranteed transactional messaging, and finally at the transactional messaging solution of RocketMQ, a modern messaging component.

Using database tables as message queues

If your application is using a relational database, the transactional outbox pattern, Transactional Outbox, can be used directly to ensure that transactions between data updates and messages are sent.

This pattern uses database tables as temporary message queues. As shown in the figure above, the service sending the message has an OUTBOX data table. When performing INSERT, UPDATE, and Delete business operations, a message record will also be inserted into the OUTBOX data table. This can ensure atomicity because this is a locally based ACID transaction.

The OUTBOX table acts as a temporary message queue, and then we introduce a Message Relay service that reads data from the OUTBOX table and publishes messages to message components.

The implementation of message relay can be very simple, just need to pull the latest unpublished data from OUTBOX table regularly through scheduled task, send the data to message component after obtaining the data, and finally delete the sent message from OUTBOX table.

Publish events using the transaction log

Another way to ensure transactional messaging is database-based transaction logging, known as Change Data Capture (CDC).

Generally, databases record transaction logs when data changes, such as MySQL binlog. The transaction log can simply be understood as a file queue local to the database, which records changes to database tables that occur in chronological order.

Here we use the alibaba open source component canal in conjunction with MySQL to illustrate how this model works.

For more instructions, please refer to the official document: github.com/alibaba/canal

Canal works

canal simulates MySQL slave interaction protocol, disguises itself as a MySQL slave node, and sends dump protocol to MySQL master;

MySQL master receives dump request and starts pushing binary log to slave (i.e. canal );

Canal parses the binary log object (originally a byte stream), and then the parsed data can be sent directly to the message component.

RocketMQ Transaction Messaging Solution

Apache RocketMQ already supports distributed transaction messages in version 4.3.0. RocketMQ uses the 2PC idea to implement commit transaction messages, while adding a compensation logic to handle two-phase timeout or failure messages, as shown in the following figure.

RocketMQ transaction message implementation is mainly divided into two stages: normal transaction sending and submission, transaction information compensation process.

The overall process is:

Normal transaction send and commit phase

1. Producer sends a semi-message to MQServer (semi-message refers to a message that consumers cannot consume temporarily)

2. The server responds to the message writing result, and the semi-message is sent successfully.

3. Start executing local affairs

4. Perform Commit or Rollback operations according to the execution state of the local transaction

Compensation process for transaction information

1. If MQServer does not receive the execution status of the local transaction for a long time, it will initiate an operation request to confirm the callback to the producer.

2. After receiving the confirmation callback request, the producer checks the execution status of the local transaction.

3. Perform Commit or Rollback operation according to the checked result

Compensation phase is mainly used to solve the producer in sending Commit or Rollback operation timeout or failure.

When the producer uses RocketMQ to send transaction messages, we will also learn from the first scheme, that is, build a transaction log table, and then generate a transaction log record at the same time when executing local transactions, so that local transactions and log transactions are in the same method. At the same time, add @Transactional annotation to ensure that the two operations are an atomic operation. In this way, if there is information about this local transaction in the transaction log table, it means that the local transaction is successfully executed and requires Commit. On the contrary, if there is no corresponding transaction log, it means that the local transaction is not successfully executed and Rollback is required.

"Java development microservices architecture how to design message queues" content is introduced here, thank you for reading. If you want to know more about industry-related knowledge, you can pay attention to the website. Xiaobian will output more high-quality practical articles for everyone!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.