Shulou(Shulou.com)06/01 Report--

Recently, in the process of learning MongoDB, I came across a new term-idempotent.

Let's first look at Du Niang's explanation.

idempotent

Idempotent (idempotent) is a mathematical and computer science concept commonly used in abstract algebra.

In programming, the characteristic of an idempotent operation is that any number of executions has the same effect as one execution.

An idempotent function, or idempotent method, is a function that can be executed repeatedly with the same parameters to obtain the same result. These functions do not affect the state of the system, and there is no fear that repeated execution will cause changes to the system. For example, the "setTrue()" function is an idempotent function whose result is the same no matter how many times it is executed. More complex idempotent operations are guaranteed using unique transaction numbers (serial numbers).

Do you understand? This explanation is more abstract, don't worry. Let's see what an idempotent problem is.

idempotent problem

The so-called idempotent, simply put, is that the result of multiple calls to the interface is consistent with the call once. By extension, the interface here can be understood as an HTTP interface or other interface that is published externally, or an internal interface that receives messages, or even an internal method or operation.

So why do we need interfaces to be idempotent? Consider the following scenario:

When placing an order in the App, after clicking OK, there was no response, so he clicked several times. In this case, if the idempotent nature of the interface cannot be guaranteed, the duplicate order problem will occur. Message push repeats when messages are received. If the interface processing the message is not idempotent, the impact of consuming messages repeatedly can be significant.

In a distributed environment, the network environment is more complex. Due to front-end operation jitter, network failure, message duplication, slow response speed, etc., the probability of repeated calls to the interface is greater than in a centralized environment, especially repeated messages are difficult to avoid in a distributed environment.

In a distributed environment, some interfaces are inherently idempotent, such as query operations. Some modifications to the data are constant and have no other records or operations, which can also be said to be idempotent. In other cases, all changes involving modifications to the data or changes in state are necessary to prevent repetitive operations from occurring. It is an effective solution to avoid the influence of repeated operations by indirectly implementing the idempotent of interfaces.

The App example above makes it easier to understand. How is idempotency guaranteed in concurrent systems?

How to ensure idempotent data in high concurrency system

In the process of system development, we often encounter problems such as repeated data insertion, repeated update, message retransmission, etc., because the complex logic of the application system and the uncertainty of network interaction will lead to this repetition phenomenon, but some logic needs to have idempotent characteristics, otherwise the consequences will be more serious, such as repeated order creation, which will bring extraordinary problems.

Idempotent property of the system

An idempotent is a concept in data that indicates that the Nth transformation and the 1st transformation have the same result.

2. How to ensure idempotent property of highly concurrent systems

1. Inquiry

Query API, can be said to be natural idempotent, because you query once and query twice, for the system, there is no change in any data, so, query once and query many times the same;

2. MVCC scheme

Multi-version concurrency control, update with condition update with condition, this is also in the system design, reasonable choice of optimistic lock, through version or other conditions, to do optimistic lock, so as to ensure timely update in the case of concurrency, there will be no big problem.

For example, update table_xxx set name=#name#,version=version+1 where version=#version# , or update table_xxx set quality=quality-#subQuality# where quality-#subQuality# >= 0

3. Separate de-duplication table

If there are many places involved in deduplication, for example, there are various business documents in the ERP system, and each business document needs to be duplicated, at this time, you can create a separate deduplication table. When inserting data, insert the deduplication table, and make use of the unique index characteristics of the database to ensure unique logic;

4. Distributed locks

Or take the example of inserting data. If it is distributed system, it is difficult to construct unique index. For example, the unique field cannot be determined. At this time, distributed lock can be introduced. Through a third-party system, data can be inserted or updated in the business system, distributed lock can be obtained, and then operation can be performed, and then lock can be released. In this way, the idea of multi-thread concurrent lock can be introduced into many systems, that is, the solution idea in distributed system;

5. Delete data

Delete data, only the first deletion is the real operation data, the second or even the third deletion, directly return success, so as to ensure the idempotent;

6. Insert unique index of data

The uniqueness of the inserted data can be constrained by the business primary key. For example, for a specific business scenario, three fields must determine the uniqueness. Then, a unique index can be added to the database table to indicate it.

Here is a scenario, idempotent at the API level, such as submitting data, how to control repeated submissions, where you can add a unique identifier to the form of submitted data or client software, and then the server will de-duplicate according to this UUID, so that you can better achieve unique identification at the API level.

By looking at this, you may already have a deep understanding of idempotencies. Finally, let's look at idempotencies in MongoDB.

Idempotent in MongoDB

MongoDB is a document-oriented non-relational database written in C++ language (it is a NoSql database implementation). It is also a data storage product between relational database and non-relational database. It provides high-performance, high-availability, high-scalability and distributed storage-based database. It is the database with the richest functions and the most similar relational database.

In MongoDB, document operations do not support transactions, but their preservation, modification and deletion of documents are atomic, that is, either the operation is completed or the operation is not completed, and there is no intermediate uncertainty, so the integrity of the data can be guaranteed.

MongoDB officially provides a method of submission in two phases. The basic principle is to use the idempotent nature of write operations, but there is a prerequisite: in the process of business implementation, the inconsistency of intermediate states can be ignored, but the final result is consistent. In fact, most requirements can be achieved as long as the results are consistent. MongoDB is also expected to do better expansion and optimization in this regard.

1. Data model

For example, but we need to add a counter field for the document, it has no actual business role, just used as an atomic flag bit avaliable, the specific code is as follows:

{

"_id" :ObjectId("57e89964b316d2e13cc0ba9b"),

"username" :"marky@123.com",

"nickname" : "marky",

"address" : "Conan Private Equity Building, 1024 Yunduan Road",

"contact" :"13141250012",

"created" : "2012-07-07",

"orders" : [

ObjectId("57e89b3ab316d2e13cc0ba9c"),

ObjectId("57e89bcfb316d2e13cc0ba9d")

],

"available": 1

}

Note:

Although this method of adding a counter flag bit can achieve atomicity, it is not friendly to business compactness and understandability, so in fact, we only use it when operating on sensitive documents, such as monetary transactions.

2. How to operate

How does it work? In fact, it can be implemented by using findAndModify() method. If the counter available value of the found document is not-1 but greater than 0, it means that there is a new operation state, and then update the new operation, and simultaneously modify available to the default value, as follows:

>db.user.findAndModify({query:{_id:ObjectId("57e89964b316d2e13cc0ba9b"), available:{$gt:0}},update:{$inc:{ available:-1},$set:{created:'2012-07-08'}}})

Results returned:

{

"_id" : ObjectId("57e89964b316d2e13cc0ba9b"),

"username" :"marky@123.com",

"nickname" : "marky",

"address" : "Conan Private Equity Building, 1024 Yunduan Road",

"contact" :"13141250012",

"created" :"2012-07-08",

"orders" : [

ObjectId("57e89b3ab316d2e13cc0ba9c"),

ObjectId("57e89bcfb316d2e13cc0ba9d")

],

" available" : 0

}

~~~~~~~ the end~~~~~~~~~

hoegh

2017.04.05

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.