Shulou(Shulou.com)05/31 Report--

This article mainly explains "what are the pits for the separation of database reading and writing". The content of the explanation in the article is simple and clear, and it is easy to learn and understand. let's study and learn "what are the pits for the separation of database reading and writing"?

Preface

The thing is, when I first started, I received a business requirement like this:

When each payment channel fails, a specific error code will be returned. Within the business, the channel-specific error code needs to be translated into an internal error code, so that our own error codes can be returned externally.

In fact, this requirement is not difficult, and the system architecture designed at that time is as follows:

The process of adding new rules is simply divided into three steps:

Business personnel add mapping rules by managing the backend

Add and modify this mapping rule to the database

Delete cach

Caching is added here because every payment in this scenario needs to be used. Using caching can avoid going to the database to read each time and increase the reading speed.

The business process of subsequent payment requests is as follows:

Database read-write separation-user actions

When the mapping rule in the cache does not exist, the database is queried and then loaded into the cache. If the in-cache mapping rule already exists, the in-cache mapping rule will be used directly.

This business process is actually relatively simple, and there was no problem testing in the test environment at that time, but the subsequent release of the online environment encountered strange problems.

"after adding the rule, the mapping rule did not take effect for a period of time. Check the log and find that when querying the database, there is no data. "

It's strange that the log shows that the addition is successful, but the query has no data. But after a period of time, the query again has the data.

I checked the code and found that there was no problem. When I went to work the next day, I consulted my colleagues before I knew the cause of the problem:

The original online database adopts master-slave architecture, data reading and writing are separated, and data query takes the slave database. Data writing is to directly operate the master library, and then synchronize to the slave library.

"due to the delay in database synchronization, during this period of data synchronization, the master and slave data will be inconsistent and the slave database cannot query the latest data. "

If your previous database system architecture is a single database or active / standby architecture, when you first switch to the data read-write separation architecture, the probability of this hole will also be stepped on.

Development of database system architecture

Let's first take a look at the database system architecture, and finally look at how to solve the data inconsistency caused by the master-slave synchronization delay.

Active and standby architecture

In the early stage of business development, the amount of data access is small, so we can directly adopt the single-database architecture.

However, we generally do not use the above architecture because of the single point problem. If the database fails, the business will be unavailable during this period. We have no other solution but to wait for a reboot.

So we will add a backup database to synchronize the data of the main database in real time.

Active and standby architecture

Once the "main library" fails, manually kick the "host" off the line and change the "standby" to "host" to continue to provide services.

This architecture is easy to deploy and maintain, and business development does not need any transformation.

However, the disadvantage is also obvious, the standby library will only be enabled when there is a problem with the main database, and there is a certain waste of resources.

Master-slave architecture

With the development of the business, the number of requests is increasing, the amount of data is also increasing, the business becomes more complex, and soon the data will reach the bottleneck.

Because most businesses read more and write less, database reading is most likely to become the bottleneck of the system.

At this time we can improve the performance of reading, at this time we can adopt the scheme to increase the separation of slave instances, master-slave synchronization, data read and write.

You can see that this architecture is no different from the master / slave architecture. The main difference is that under the master / slave architecture, the slave library, like the master library, needs to work all the time. The master library provides write services, and the slave library only provides read services.

If the pressure of subsequent reading is still too great, we can also increase the number of libraries and horizontally expand our reading ability.

Although the master-slave architecture helps us solve the bottleneck of reading, due to the need for data synchronization between master and slave, there is a natural delay.

During this delay window, only one old data can be read from the library, which is the real cause of the problem in the above case.

Next, let's see what we can do to optimize this situation.

Master-slave delay solution

Endure the Dafa

The first solution is very simple, there is no other, regardless of him, did not read it is fine. At this time, the business does not need any transformation. Hello, I am good, and she is all right.

If the business does not have high requirements for data consistency, we can adopt this scheme.

Data synchronous write scheme

The master-slave data synchronization scheme generally uses the asynchronous mode to synchronize to the standby database.

We can modify it to the synchronization scheme, the master-slave synchronization is completed, and the writes on the master library can be returned.

The business system initiates the write operation, and the data is written to the main database.

The write request can not be returned until the master-slave synchronization is completed.

The data is read from the slave database, and the latest data can be read after the master-slave synchronization is completed.

In this scheme, we only need to modify the synchronization configuration between databases, and the business layer does not need to be modified, which is relatively simple.

"however, because master library writes need to wait for master and slave to complete, the latency of write requests will increase and the throughput will decrease. "

This may not be acceptable to the current online business.

Selective compulsory reading

For scenarios that require strong consistency, we can operate all read requests on the main library, so that "both reads and writes are in the main library" and there are no inconsistencies.

The business layer of this scheme needs to be reformed to make it mandatory, which is relatively difficult to transform.

However, this scheme increases the pressure on the main database rather than wasting another database.

Middleware selection routing method

This scheme requires the use of a middleware in which all database operations are first sent to the middleware and then distributed to the corresponding database.

The process is as follows:

Write the request, and the middleware will send it to the main library, and record the key of the write request at this time (operation table plus primary key, etc.)

Read request. If key exists at this time, it will be routed to the main library.

After a certain period of time (empirical value), the middleware thinks that the master-slave synchronization is completed, delete the key, and the subsequent reading will read the slave library.

This scheme can maintain the consistency of data reading and writing.

However, with the addition of a middleware in the system architecture, the overall complexity becomes higher, the business development becomes complex, and the learning cost is relatively high.

Cache routing Dafa

This scheme is similar to the scheme flow of middleware, but the transformation cost is relatively low and no middleware is needed.

The process is as follows:

The write request is sent to the master database, and the key of the record operation is cached, and the cache expiration time is set to the master-slave delay.

The read request first determines whether the cache exists.

If it exists, it means that a write operation has just occurred, and the read request operates on the main database

If it does not exist, it means that no write operation has occurred recently, and the read request operation is performed from the database.

The cost of this scheme is lower than that of middleware, but we introduce a cache component at this time, which adds one more cache operation between all reads and writes.

Thank you for your reading. The above is the content of "what are the pits of database reading and writing separation". After the study of this article, I believe you have a deeper understanding of the problem of database reading and writing separation pits. The specific use of the situation also needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.