Shulou(Shulou.com)06/01 Report--

This article mainly explains "what is the method of database data migration based on account system". The content of the explanation in this article is simple and clear, and it is easy to learn and understand. let's study and learn "what is the method of database data migration based on account system"?

1. Database Analysis of Old and New Systems

The new and old systems are all based on Mycat shards, the old system has 12 shards, and the new system has 8 shards. Because the new system is reconstructed, only one of the account types of members (member main wallet, equivalent to Yu'e Bao) is reconstructed, and the remaining fruit currency, points and coupons are slowly reconstructed in the following iterations.

If you count the main wallet alone, the user information is about 50 million, the main wallet account information is about 50 million, and the flow is about 80 million. There are three tables involved, called user_info and account_ information tables and account_ record tables, which correspond to these two or three tables in the new system. The difference is that the data inside needs to be converted. As shown in the figure, the old system is based on old_user_id fragmentation, while the new system is based on new_user_id fragmentation.

two。 The threading model adopted and the way in which the 2.1 program connects to the database

Both systems are shredded based on Mycat. If the migration program connects to the database through mycat, the speed will be slightly slower. It is intended to migrate by directly connecting each shard. However, due to the different number of shards and the different shard fields on both sides, you need to write your own shard rules when migrating to the new system through direct connection.

As shown in the directly connected way, when reading a batch of data from the old system, it is necessary to calculate which shard falls into the new system. In order to reduce the implementation complexity and quickly locate the problems that occur in the formal migration, another method is adopted. In the old system, 12 slices are directly connected, which are written to the new system through mycat, and the calculation of rules is handled by Mycat.

2.2 through queuing, multi-thread reading, multi-thread writing

In order to improve the speed, I did not hesitate to choose the multi-threaded scheme. At the beginning, I thought of the queue-based way, reading and writing at the same time; the old system had a total of 12 shards, and each time multiple threads started to read one shard of data into the blocking queue, while multiple threads took data from the queue and assembled it and then wrote it to the new system through Mycat. Until the data of one shard is read, the next shard is read.

Advantages: fast reading speed

Disadvantages: slow writing, resulting in message accumulation, insufficient memory, frequent GC and slow speed.

2.3 the thread is cut by the dimension of the fragment, and the same thread reads and writes synchronously (take reading user information and account information as an example)

Since the migrated server is 32 gigabytes with 8 cores, only 8 threads are opened to read 12 shards of data, and read and write data synchronously in the same thread. As shown in the figure, the steps are as follows. Based on this scheme, the migration time of user and account data is about 26 minutes.

Put 12 shards of the old system into the blocking queue.

Each thread starts with a sharding data source from the queue, through a direct connection

Read in pages, about 5000 pieces per batch, assemble the data and insert the data through the mycat of the new system.

Until the last batch of data is less than 5000 pieces, it is considered that the data of the fragment has been read.

After reading, the thread switches parts, that is, goes back to the queue to get the new part. If the part of the queue is empty, the thread exits, otherwise repeat 1-4.

Thank you for your reading. the above is the content of "what is the method of database data migration based on account system". After the study of this article, I believe you have a deeper understanding of what the method of database data migration based on account system is, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.