Shulou(Shulou.com)06/01 Report--

Background

In the process of rapid iteration of the system, the business system is often deployed in the same physical library without physical isolation between core data and non-core data. With the expansion of the amount of data, this situation will bring the risk of stability, such as the slow sql of the library, disk, IO and so on, which will affect the business stability of the core system as a whole. Therefore, it is necessary to extract the business tables of the core business from the original database and separately into the new database. The migration of core data involves a key difficulty: how to migrate data smoothly and without user perception. Combined with Xianyu's commodity library migration practice, this paper will show you how to solve this problem.

Current situation of Xianyu's Commodity data

Xianyu uses sub-table and sub-database and MYSQL database cluster to support online query service. As shown in the figure below, it is managed efficiently and uniformly through TDDL [1] database middleware. Some students may not understand the concepts related to sub-table and sub-library, so here is a brief introduction.

01 sub-table and sub-database principle

In essence, it is a problem of horizontal splitting of a database. A database is divided into multiple parts and placed on different databases (server), thus alleviating the performance problems of a single database. The following figure describes the core principles of sub-table and sub-database:

Of course, sub-table sub-database also has a negative impact, that is, table structure changes and related management compared with single table trouble, there are certain risks, specific how to choose, or according to the actual situation to analyze.

Global Sequence generation under 02 sub-table and sub-library

Sub-table and sub-library solve the problem of online service capacity and performance, but it also brings about an increase in complexity. Flexible configuration of routing rules and routing data and the provision of easy-to-use encapsulation are considered so that the business is not aware of it. Ali open source middleware product TDDL provides a solution, corresponding to Ali Cloud products: DRDS [2].

The key principles of TDDL are not introduced much, but the risk of primary key conflict is an important risk point of failure in the process of database migration. Here is a brief introduction to the global unique primary key generation principle of TDDL.

As shown in the figure above, TDDL Sequence is based on database update and memory allocation: each operation allocates id in batches, and the number of id allocated is the internal step of sequence. The original id value is added with the external length value, and the subsequent allocation is directly taken in memory. This advantage: simple and efficient disadvantage: cannot guarantee the self-increment order.

In addition, in the process of data migration, in order to ensure that there is no conflict with the primary key of the original database, it is necessary to set a primary key with a larger transition to prevent the primary key conflict between the two libraries, which is one of the key points that should be paid attention to in the follow-up migration.

Data migration scheme

Through the brief introduction of the previous article, we have a preliminary understanding of the current situation of Xianyu commodity storehouse. Here we will introduce how Xianyu has steadily migrated the commodity storehouse.

01 core ideas

The core idea of data migration is actually very simple, that is, how to migrate data stably and smoothly, as shown in the following figure:

However, if we refine this process, we will encounter many problems, such as:

1. How should we migrate the data, which is one-time? Or in stages?

2. How to verify the correctness of the data migration process?

3. What should we do if we have problems in business transformation? How to find out as soon as possible? How to roll back?

4. How is the performance of our new library?

With these questions, let's sort out the migration plan in detail.

02 implementation scheme

As shown in the figure above, the whole solution is divided into several parts:

1. System transformation, including SQL transformation, double write switch, new library sequence creation.

SQL transformation: load two sets of TDDL data sources, one for the old library and one for the new library, and generate two sets of mybatis sql templates.

Double write switch: set the switch to write the new library and write the old library, which is used for the double write process in the process of online migration and timely rollback in case of problems.

Sequence creation: when migrating sequence tables, you need to raise the values in the sequence table of the new library to prevent primary key conflicts, and you need to evaluate a safe value according to primary key consumption, which is a very important detail, which is emphasized again.

2. stability guarantee, relocation of storage is a major event, and stability is the most important in the process of transformation, including system pressure measurement, online flow playback, and fault drills.

System pressure test: mainly for the performance test of the new library to prevent unexpected conditions in the new library.

Online traffic playback: Edsger W. Dijkstra said that if debuggers are a standard process that can eradicate BUG, then programming is the process of putting them in. Through the introduction of online data playback in the test environment, problems can be found as much as possible to ensure the stability after the transformation.

Fault drill: by injecting some man-made faults, such as failure to write a new library, there is something wrong with the logic of the new library, and rehearse the rollback strategy in a timely manner.

3. Data migration mainly takes advantage of the data migration capability of Aliyun data transfer service DTS [3], which involves full migration, incremental migration, consistency verification and reverse tasks.

Full migration: the primary goal of data migration is how to migrate all historical data to the new database. Our approach is to specify a time point, find the maximum and minimum id of each source table, and then import them to the target database in batches, as shown in the figure:

* the whole process is to query the preparation of the online database, so it does not affect the database service of the online business.

Incremental migration: since the business service is running all the time during the migration, the full migration is complete, and the data after the full time point should be recovered. The core principle here is to synchronize the binlog log data after the full time point to ensure data consistency. It should be noted that the incremental time needs to be moved forward by a short break time (such as 5 minutes). The main reason is that there is a time difference at the moment when the full migration starts. Incremental forward is required to ensure the final consistency of the data, as shown in the following figure:

Consistency check: after full and incremental migration, the source database is theoretically consistent with the target data, but in practice, it is applied in the stages of functional testing, online traffic playback and so on. The data may be inconsistent in this process, so data consistency verification needs to be done before it is officially launched. The principle is to query the source table in batches (similar to the query method of full migration). Then compare it with the target library, as shown in the figure:

Reverse task: after migrating to the new database, the offline business still depends on the old database. You need to set up a backflow task from the new database to the old database. The principle is the same as incremental migration, only changing the original database and the target database.

03 relocation process

At this point, we should be relatively clear about the points involved in the relocation, but there is also a very important thing, that is, it is very critical to sort out the whole relocation steps, and there are usually two plans.

Option 1:

1. DTS data leveling, that is, full synchronization is completed, incremental synchronization is enabled, and the delay is within seconds.

2. Check before going online, mainly replay the flow on the cable, pressure test, consistency check, fault drill.

3. Open double write online, and write both the new library and the old library online. At this time, you need to turn off the incremental synchronization task to prevent invalid overwriting.

4. Online verification, execute the pre-prepared test script and combine the consistency verification tool, and slowly cut the read traffic to the new library to verify the double-write logic.

5. Switch the data source, turn off double write and formally write to the new library.

6. Create a reverse task and flow the data back to the old library.

Option 2:

1. DTS data leveling, that is, full synchronization is completed, incremental synchronization is enabled, and the delay is within seconds.

2. Check before going online, mainly replay the flow on the cable, pressure test, consistency check, fault drill.

3, online switch, write new library, at the same time need to turn off incremental synchronization task to prevent invalid overwrite.

4. Create a reverse task and flow the data back to the old library.

Comparison of advantages and disadvantages of solutions:

To sum up, the migration process of scenario 1 is relatively complex, the control of migration is more detailed, it is suitable for business complexity, and there are many underlying transformations, and you want to finely control the scenarios where you want to control the migration steps. Scenario 2: migration is relatively simple and the process is fast. It is suitable for scenarios where the business process is relatively simple and controllable. Students can choose which solution to choose according to their own business conditions.

Here, taking into account the complexity of Xianyu commodity business, more transformation at the bottom, from the point of view of stability, the final choice of scheme 1.

In option 1, the most important steps are 3, 4 and 5, so you need to plan the rollback in advance.

04 rollback scheme

The general principle of the rollback scheme is not to lose data. The most likely occurrence point is that there is a problem with the new database during double writing, resulting in an exception in the online service. In this case, you only need to close writing to the new library immediately, and if there is a problem with the new library (such as performance problems), you can immediately switch back to the old library and maintain data consistency through reverse tasks. Finally, if distributed transactions are not enabled, the shorter the time for double writing, the better, and there may be data inconsistencies.

Summary

Through careful migration scheme design and the ability of DTS's powerful data migration tools, Xianyu commodity library successfully completed the migration of XXBN online database services, and independent physical deployment significantly improved the stability of online services in the commodity library. However, the database situation of different business systems may be different, such as migration from single database to multi-database, migration from single table to multi-table, etc., but the overall scheme is roughly similar. I hope that the practical scheme of database migration in this paper can provide you with a feasible reference.

references

[1] TDDL open source Github address https://github.com/alibaba/tb_tddl

[2] https://www.aliyun.com/product/drds, the official website of Aliyun DRDS products

[3] https://www.aliyun.com/product/dts, the official website of Aliyun DTS products

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.