Shulou(Shulou.com)06/02 Report--

This article mainly explains "how to achieve idempotency in distributed systems". The content of the explanation in this article is simple and clear, and it is easy to learn and understand. let's study and learn "how to achieve idempotency in distributed systems".

Case 1: money transfer system

In the previous article, the case of money transfer system has been mentioned many times, because this case is so typical that many universities use this case when teaching database transactions.

For a single application version of the money transfer system, we can directly use the transaction of the database to ensure the ACID of the whole transfer operation. However, with the increase of the level of users, the bottleneck of a single database also appears, so the design of sub-database and sub-table appears, that is, part of the user information is stored in one database, and the other part is stored in another database. Based on this design, transactions in a single database must not be available. We need to use distributed transactions across databases, such as distributed transactions based on XA protocol, but this approach has its own problems and limitations of application scenarios. Therefore, generally speaking, the actual scenario is to adopt the final consistency solution based on BASE.

The following is a simple final consistent scheme design:

After receiving a transfer request from the user, Step 1:Application first executes the logic of the transferor, as follows:

Begin transaction billing (including: transfer request uuid+ transfer status in progress) deduction (balance reduction of transferor) commit/rollback

This logic is contained in a transaction, and since only one database is involved, you can take advantage of the transaction guarantee of a single database.

Step 2: a background job constantly grabs in progress's bill, then sends event (notify the cashier to collect money) to Kafka, and changes the bill status to success after it is successfully sent.

This logic is the implementation of outbox pattern. For a specific introduction to outbox pattern, you can refer to my other article (talk about Outbox Mode (outbox pattern)).

Step 3: the transferor has a listener listening to the event all the time. When the monitor hears the event, execute the following logic:

Begin transaction billing (including: transfer request uuid+ transfer status success) plus money (transferer balance increase) commit/rollback

The logical processing of the transferor is also in a transaction, which can be guaranteed by the transaction of a single database.

However, the above design may resend event messages in several places, such as: background job succeeded in sending event, but failed to change the billing status; or, the transferor logic commit to the database was successful, but there was a problem sending ack to Kafka, and so on. So, how to deal with such repeated consumption of messages? Because if not handled properly, it may lead to data inconsistency. In fact, this is essentially an idempotent problem, ensuring that the processing results of receiving repeated messages and receiving a message are consistent, that is, idempotent.

For the above design, to ensure idempotency, you can store a request uuid in the billing table and use this uuid to achieve the effect of removing duplicates. Specifically, when the transferor receives the repeated transfer event message, it first goes to the database to check whether the ID exists according to the request uuid, or indicates that the transfer has been processed and the event is ignored directly. If not, it means that the event needs to be processed and the transfer is performed. Generally speaking, such processing logic is idempotent.

Of course, the actual transfer system also needs to consider a variety of error situations, such as: if the transfer party fails to process, it can send a reverse event, and the outgoing party can return the previous deduction revert.

Case 2: data migration

The case of data migration has also been mentioned many times in previous articles. This case is about the need to migrate data from the old database to the new database, and to ensure that the service does not stop (zero downtime), that is, does not affect the normal use of users.

For old data, you can directly use a background job for continuous migration; the key is how to "migrate" new data? One way is to double write, that is, to write to the old database as well as to the new database, so as to ensure that the new data is available on both sides.

If you write to two databases at the same time, how can you ensure that both sides succeed and fail? This is also the problem of distributed transactions, when a solution was mentioned: best effort 1pc, which uses ChainedTransactionManager provided by Spring. However, this approach can also be inconsistent in extreme cases, such as database downtime at a specific time node.

Here's another event-based double write: after writing the data to the old database, the data itself is sent to Kafka as event payload. (here you can use outbox pattern to guarantee at least once delivery.) then, add a new piece of logic, listen to the event, receive the event, and write the data to the new database.

Similarly, in the case of snooping event, additional handle is required to ensure idempotency:

Received repeatedly inserted data event (this situation is similar to the case of transfer above)

In this case, how to achieve idempotent processing?

Similarly, you can rely on a unique primary key to determine whether the data exists or not.

Message order change

There is a change in the order of messages, possibly in the following situations:

-retry queue, which updates the event of the same data twice in a row. The first event processing fails to put into retry queue, while the second event processing succeeds. -when traffic is cut to a new database, there is an event for updating data in Kafka. At this time, a request for updating data has come in.

In this case, how to ensure idempotency?

The key point is that the old event needs to be ignored. The implementation plane can rely on a timestamp, whether it is the migration data itself or the event object itself. If the new event has been processed, the old event is ignored; if the data has been updated, the old event is ignored.

The double write mentioned above requires an additional event database table. If you can, you can also use cdc, which is often used in database replication, backup, and other scenarios. In this way, you do not need to write an additional table, but rely on the transaction log of the database. For more information, please refer to my other article (talk about Outbox Mode (outbox pattern)).

Thank you for your reading. the above is the content of "how to achieve idempotency in distributed systems". After the study of this article, I believe you have a deeper understanding of the problem of how to achieve idempotency in distributed systems. the specific use situation still needs to be verified by practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.