Shulou(Shulou.com)06/03 Report--

This article introduces the relevant knowledge of "how to ensure the idempotence of the interface". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Preface

I don't know if you've ever encountered these scenes:

Sometimes when we fill out some form forms, the Save button accidentally clicks quickly twice, and there are two duplicate pieces of data in the table, but the id is different.

In order to solve the problem of interface timeout in the project, we usually introduce a retry mechanism. The first request interface timed out, and the requester failed to get the returned result in time (it may have been successful at this time). In order to avoid returning an incorrect result (in this case, it is impossible to return a failure directly?), the request will be retried several times, which will also produce duplicate data.

When mq consumers read messages, they sometimes read duplicate messages (not to mention the reason here, interested partners can talk to me in private), and duplicate data will be generated if they are not handled properly.

Yes, these are all idempotent problems.

Interface idempotency means that the result of one or more requests initiated by a user for the same operation is consistent, and there are no side effects caused by multiple clicks.

Most of these problems are caused by interfaces:

Insert operation, in which case multiple requests may result in duplicate data.

Update operation, if it is simply to update the data, such as update user set status=1 where id=1, there is no problem. If there are still calculations, such as update user set status=status+1 where id=1, multiple requests in this case may result in data errors.

1. Select before insert

Usually, in the interface for saving data, in order to prevent duplicate data, we usually select the data according to the name or code field before insert. If the data already exists, the update operation is performed, and if it does not exist, the insert operation is performed.

This scheme is probably the one we use most often to prevent duplicate data. However, this scheme is not suitable for concurrent scenarios. In concurrent scenarios, it should be used with other schemes, otherwise duplicate data will also be generated. I would like to mention it here in order to prevent everyone from stepping in the pit.

two。 Add pessimistic lock

In the payment scenario, the account balance of user An is 150 yuan and wants to transfer 100 yuan. Normally, the balance of user An is only 50 yuan. In general, sql looks like this:

Update user amount = amount-100 where id=123

If the same request occurs multiple times, it may cause the balance of user A to become negative. In this case, user A may cry. At the same time, system developers may also want to cry, because this is a very serious system bug.

To solve this problem, you can add a pessimistic lock to lock user A's row of data, allow only one request to acquire the lock at a time, update the data, and other requests wait.

Typically, a single row of data is locked with the following sql:

Select * from user id=123 for update

The specific process is as follows:

Specific steps:

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

Multiple requests query user information according to id at the same time.

Determine whether the balance is less than 100, and if the balance is insufficient, return the insufficient balance directly.

If the balance is sufficient, query the user information again through for update and try to acquire the lock.

Only the first request can acquire the row lock, and the rest of the requests that do not acquire the lock wait for the next opportunity to acquire the lock.

After the first request acquires the lock, it determines whether the balance is less than 100, and if the balance is sufficient, the update operation is performed.

If the balance is insufficient, it means that the request is repeated, and the success is returned directly.

It is important to note that if you are using an mysql database, the storage engine must use innodb because it supports transactions. In addition, the id field here must be a primary key or a unique index, otherwise the entire table will be locked.

Pessimistic lock needs to lock a row of data during the same transaction operation. If the transaction takes a long time, it will cause a large number of requests to wait and affect the performance of the interface. In addition, it is difficult to guarantee the same return value for each request interface, so it is not suitable for idempotent design scenarios, but it can be used in anti-repetition scenarios. By the way, there is a difference between anti-weight design and idempotent design. Anti-redesign is mainly to avoid duplicate data, there are not too many requirements for interface return. The idempotent design not only avoids duplicating data, but also requires that each request return the same result. 3. Add optimistic locks since pessimistic locks have performance problems, in order to improve interface performance, we can use optimistic locks. You need to add a timestamp or version field to the table. Take the version field as an example.

Before updating the data, query the data:

Select id,amount,version from user id=123

If the data exists, assume that the version found is equal to 1, and then use the id and version fields as query criteria to update the data:

Update user set amount=amount+100,version=version+1 where id=123 and version=1

Version+1 the data at the same time, and then determine the number of rows affected by this update operation. If it is greater than 0, the update is successful. If it is equal to 0, the data has not been changed.

Since the first request for version equals 1 can be successful, the version becomes 2 after a successful operation. At this point, if the concurrent request comes, execute the same sql:

Update user set amount=amount+100,version=version+1 here id=123 and version=1

This update operation does not really update the data, and the final number of rows affected by the execution of the sql is 0, because the version has become 2, and the version=1 in the where certainly cannot meet the condition. However, in order to ensure the idempotency of the interface, the interface can directly return success, because the version value has been modified, so it must have been successful once before, followed by repeated requests.

The specific process is as follows:

Specific steps:

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

First query user information based on id, including version field

Update the user information according to the id and version field values as parameters of the where condition, while version+1

Determine that the operation affects the number of rows. If it affects 1 row, it is a request and other data operations can be done.

If row 0 is affected, it means that the request is repeated, and success is returned directly.

4. Add a unique index

In most cases, in order to prevent duplicate data, we will add a unique index to the table, which is a very simple and effective solution.

Alter table `order` add UNIQUE KEY `un_ code` (`code`)

After adding a unique index, the first request data can be inserted successfully. However, the following same request will report a Duplicate entry '002' for key 'order.un_code exception when inserting data, indicating that there is a conflict in the unique index.

Although throwing an exception has no effect on the data, it will not cause incorrect data. But in order to ensure the idempotency of the interface, we need to catch the exception and return success.

If the java program needs to catch: DuplicateKeyException exception, if you use the spring framework, you also need to catch: MySQLIntegrityConstraintViolationException exception.

The specific flow chart is as follows:

Specific steps:

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

The user initiates the request through the browser and the server collects data.

Insert this data into mysql

Determine whether the execution was successful, and if so, manipulate other data (and possibly other business logic).

If the execution fails, a unique index conflict exception is caught and a success is returned directly.

5. Build a weight prevention table

Sometimes not all scenarios in the table are not allowed to produce duplicate data, only certain scenarios are not allowed. At this point, it is obviously not appropriate to add a unique index directly to the table.

In view of this situation, we can solve the problem by building an anti-weight table.

The table can contain only two fields: id and a unique index, which can be the unique identifier of a combination of multiple fields, such as name, code, and so on, such as susan_0001.

The specific flow chart is as follows:

Specific steps:

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

The user initiates the request through the browser and the server collects data.

Insert this data into the mysql anti-repetition table

Determine whether the execution was successful, and if so, do other mysql data operations (and possibly other business logic).

If the execution fails, a unique index conflict exception is caught and a success is returned directly.

It is important to note that the anti-repetition table and the business table must be in the same database and operate in the same transaction.

6. According to the state machine

In many cases, the business table is stateful, such as: 1-order, 2-paid, 3-completed, 4-cancelled, and so on. If the values of these states are regular, according to the business node happens to be from small to large, we can use it to ensure the idempotency of the interface.

If the order status of id=123 is paid, it will be completed now.

Update `order` set status=3 where id=123 and status=2

On the first request, the status of the order is paid and the value is 2, so the update statement can update the data normally. The number of rows affected by the sql execution result is 1, and the order status becomes 3.

Later, there is the same request, and when the same sql is executed, because the order status becomes 3 and status=2 is used as a condition, the data that needs to be updated cannot be queried, so the number of rows affected by the final sql execution result is 0, that is, the data will not be really updated. However, in order to ensure the idempotency of the interface, when the number of affected rows is 0, the interface can also directly return success.

The specific flow chart is as follows:

Specific steps:

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

The user initiates the request through the browser and the server collects data.

Update to the next state according to the id and the current state as conditions

Determine that the operation affects the number of rows. If one row is affected, the current operation is successful and other data operations can be performed.

If line 0 is affected, it means that the request is repeated and the success is returned directly.

The main special note is that the scenario is limited to the table stateful fields to be updated, and the special case where the status fields are to be updated does not apply to all scenarios.

7. Add distributed lock

In fact, the previous introduction to add a unique index or anti-repetition table, in essence, the use of the database distributed lock, also belongs to a kind of distributed lock. However, due to the poor performance of database distributed locks, we can use redis or zookeeper instead.

In view of the fact that many companies switch to apollo or nacos for distributed configuration centers and seldom use zookeeper, let's take redis as an example to introduce distributed locks.

At present, there are three main ways to implement redis distributed locks:

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

SetNx command

Set command

Redission framework

Each scheme has its own advantages and disadvantages. I won't talk about the specific implementation details. Interested friends can add me Wechat to talk to me in private.

The specific flow chart is as follows:

Specific steps:

When the user initiates the request through the browser, the server collects the data and generates the order number code as the only business field.

Using the set command of redis, set the order code to redis and set the timeout.

Determine whether the setting is successful, and if the setting is successful, it means that it is the first request, then the data operation is carried out.

If the setting fails, it means that the request is repeated, and success is returned directly.

It is important to pay special attention to: the distributed lock must set a reasonable expiration time, if the setting is too short, it can not effectively prevent repeated requests. If the setting is too long, the storage space of redis may be wasted, depending on the actual business situation.

8. Get token

In addition to the above scenario, there is a final one that uses token. This scenario is a little different from all previous scenarios, requiring two requests to complete a business operation.

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

Request for token for the first time

The second request completes the business operation with this token.

The specific flow chart is as follows:

The first step is to get the token.

The second step is to do specific business operations.

Specific steps:

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

When the user visits the page, the browser automatically initiates a request to get token.

The server generates token, saves it to redis, and returns it to the browser.

The user carries the token when initiating the request through the browser.

Query whether the token exists in redis. If it does not exist, it is the first request, and the subsequent data operation will be done.

If it exists, it means that the request is repeated, and the success is returned directly.

In redis, token is automatically deleted after the expiration time.

The above scheme is designed for idempotency.

If it is a weight-proof design, the flow chart should be changed:

It is important to note that token must be globally unique.

This is the end of the content of "how to ensure the idempotency of the interface". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.