Shulou(Shulou.com)06/02 Report--

On January 4, 2019, OceanBase migration service solution was officially launched at ATEC City Summit. Ant Financial senior technical expert Shi Wenhui and technical expert Han Guyue shared the important features and business practices of OceanBase migration service.

Three generations of ant database architecture

Over the past decade, ants have gone through three generations of upgrades across the entire underlying database architecture. The first generation of data architecture was built on IOE-IBM's minicomputer, Oracle's commercial database, and EMC's shared storage. The operation and maintenance costs based on the first generation IOE architecture are very high, and the stability challenges are also very large. With the rapid development of the business, this set of architecture has no way to adapt to the growth rate of business development.

The second generation architecture was born. The main body of the second generation architecture was OE-Oracle and EMC, plus Ant's own distributed middleware, which solved the horizontal and vertical elasticity of the business. This generation of architecture actually accompanied ants for many years.

With the arrival of 4G and 5G era and the popularization of finance, people's life is more and more inseparable from mobile payment, and the development of business spurt puts forward higher requirements for the underlying database. These requirements include greater stability, rapid recovery and extreme resilience.

And that eventually led to the third generation architecture that we have today. The third generation architecture consists of a financial cloud database represented by OceanBase and distributed middleware.

Challenges of database architecture upgrades

With the development of the whole ant, the architecture of the whole database has only evolved for three generations. One of the important reasons for this is that for any enterprise, upgrading the entire database architecture is a very challenging thing.

Shi Wenhui, a senior technical expert at Ant Financial Services, said,"To use a metaphor we often say internally, the upgrade of the database architecture is like replacing the engine for a high-speed aircraft. "

The purpose of changing engines is to have better power and do more technical innovation. However, the problem in front of us is how to innovate steadily and ensure the smooth and smooth operation of the aircraft in flight. This is actually a very big challenge.

In the evolution of the past three generations of architecture, we can see that in essence, the iteration of each generation of architecture is basically a cycle of two to three years, which will have very high labor investment and cost.

The second challenge was how to ensure stability during and after the migration from a traditional commercial database to OceanBase.

Another big challenge is data quality. In financial enterprises, data carries not only money, but also the trust of hundreds of millions of users. Therefore, one piece of data cannot be lost and one cannot be wrong. This is the bottom line of our database.

Of course, compatibility issues and performance risks also pose challenges to database architecture upgrades.

OceanBase Migration Services: A Direct Path to a Distributed Architecture Upgrade

Based on the above problems and challenges, and after ten years of advanced experience in database architecture upgrade, Ant Financial Services has created this one-stop data migration solution-OceanBase Migration Service (OMS) for customers.

Evolution of OMS

The evolution of OMS is business-driven and is inextricably linked to OceanBase's architectural upgrades and continuous development.

As early as 2014-2015, some core businesses on Ant Main Station, including well-known transaction business, payment business and membership business, need to be migrated from Oracle to OceanBase. OMS at the time supported these projects in a tool-like, modular form.

So in 2015 we started a comprehensive survey of OMS solutions, striving to precipitate common systematic solutions.

In 2016, OMS already has a platform-based architecture, introducing the idea of large-scale orchestration, integrating the whole migration process, especially the cumbersome and error-prone links, into the platform. During this period, OceanBase also completed the architecture upgrade from version 0.5 to version 1.0. In this year, OMS also supported the core business upgrade of online merchant bank, India PayTM and main station to OceanBase version 1.0.

By 2018, OMS has been polished to perfection at both the basic function level and the task scheduling level. This year OMS has supported the migration of Ant Forest, Ant Merchant Platform and a number of core and non-core businesses from MySQL to OceanBase. At the same time, external businesses, including many commercial banks that have already launched OceanBase, have also demonstrated the ability to migrate to OceanBase using OMS with one click.

OMS program advantages

The OceanBase migration service addresses five key issues.

1. Load playback verification: The first core problem is load playback verification. By collecting SQL traffic from the source database and playing it back on the target database OceanBase, you can verify whether its functions on OceanBase are compatible and whether there are performance problems. At the same time, based on more than ten years of experience of ant DBA, OMS will provide customers with tuning suggestions on performance and other aspects.

2. Second-level data verification: The second point is data verification. OMS has three layers of data verification, which can achieve second-level delay. As an example, let's say we want to replace a traditional commercial database with OceanBase, and if there is an error in any piece of data during the migration process, it can be quickly detected in one second. The delay of verification can be guaranteed to be within one second, which is about 100-200 milliseconds according to the experience on ant line.

Minute instant rollback: The third and most important point is that OMS has the ability to rollback at any time, and the rollback is lossless. This is also the cornerstone of sound innovation that we emphasized earlier.

4. Multiple database types support: At present, OMS supports Oracle, MySQL, OceanBase, etc., and supports full migration and incremental data synchronization.

5. One-click migration: The entire data migration link and rollback mechanism are basically built through one-click operation, which is easy to use.

Technical architecture of OMS

The core solution of OMS is actually very simple. We turn OceanBase into a backup library for Oracle/MySQL.

Traditional commercial databases generally have a main database and a backup database: the main database bears the write traffic, and if there is a problem with the main database, we will switch the data to the backup database, and then complete the switch through a complete set of virtual main and backup solutions provided by OMS. For example, Oracle originally had a master database and a backup database, and then OceanBase actually became a virtual backup database.

The upgrade of the entire database architecture will also become extremely simple, as simple as making a primary and secondary switch. Rollback will also become very simple, in fact, also made a main standby switch.

From the overall architecture of OMS, in fact, a very key point is that we have established a set of virtual primary and standby links between the traditional commercial database and OceanBase. All the components used in the whole OMS are actually deposited in ants and Ali for many years, and are also generated based on real scenes.

OMS migration process

The overall migration process for OceanBase migration services is actually only seven steps.

1. Evaluation: The first step is to do compatibility analysis through the load playback tool;

PoC: OceanBase cloud platform can help customers deploy a PoC cluster;

3. Pre-migration: OMS then pre-migrates Oracle data online to a test library;

4. Validation: Use the load replay tool in this test library to replay these SQL, and then find the parts of SQL that are incompatible, performance or data quality that do not meet expectations, and provide optimization suggestions;

5. Formal migration: After the first four steps are completed, the SQL that needs to be adjusted or optimized has been optimized, and then the formal migration can be completed. First of all, the original full amount of data is transferred, and then the incremental change of that part of the data real-time synchronization;

6. Verification: Wait until all the data is ready, then we continue to complete the third level verification;

7. Switch and rollback: Wait until all the checks are completed, you can complete the switch and rollback function with one key.

Complete migration from a traditional commercial database to a distributed database can be easily accomplished in these seven steps.

Ant Merchant Platform OMS-based Business Practice

Ant merchant platform carries merchant file data information, subscription relationship, contract information data and corresponding service capabilities. Part of the business uses MySQL databases, and part of the core business uses Oracle databases.

With the rapid growth of merchants and the continuous enrichment of business scenarios, the merchant platform data grows rapidly, and the data scale is quite large. Especially MySQL's single table bottleneck is becoming more and more obvious, and the performance and risk of DDL changes and DML updates can no longer be borne.

Han Guyue, a technical expert of Ant Financial Services, said,"OceanBase can support unlimited expansion of data and meet the capacity and performance requirements of merchant services. So if we change the database chassis, the performance, stability and data quality risks we face are also inevitable. "

From the business practice of ant merchant platform, comparing OMS migration with traditional migration, we can see:

·Business assessment and transformation

In the past, it usually took 1-2 months for a business to be transformed and adapted; based on the SQL compatibility assessment and load playback capabilities of OMS automation, the transformation of Ant Business Platform business took only about one week.

·Data migration and validation

Objectively speaking, the total length of migration depends mainly on the business data model, data volume and network environment. In terms of improving migration efficiency, the delay of OMS incremental migration is only milliseconds at present, and it takes only 3 seconds at most in cross-city cases. In addition, SQL and correction schemes are provided for the data differences checked, which greatly improves the overall efficiency of migration and verification.

·Service switching

In fact, before switching, it is often necessary to formulate strict flow switching scheme and Failover scheme. The details that need to be checked and verified in the whole switching process are very complicated, and any negligence may cause data inconsistency. Then OMS introduces the idea of large-scale arrangement and puts all the complicated links into the platform. Therefore, it takes 1-2 weeks to switch from the original service. After using OMS, it only takes a few minutes for the ant merchant platform business to read or write.

·Business Rollback

In the past, business rollback after migration had to bear significant decision risks. OMS makes business rollback like an active/standby switch, which can be completed instantly without losing data, so business rollback is no longer a problem. Business jitter also occurred during the overall migration of merchant business, and it only took a few minutes from login to rollback when using OMS rollback.

Therefore, the migration time of this business of ant merchant platform is about three weeks, so OMS greatly improves the overall migration efficiency of database in terms of labor cost and time cost.

Finally, Han Guyue showed you a demo demonstration of OMS one-click migration.

Today, more and more enterprises have recognized the tremendous value of distributed architecture in enabling flexible business scaling and agile development. OceanBase constantly delivers "Internet genes" to traditional enterprises through product innovation, helping more customers transform to distributed architecture.

At the same time, OceanBase is constantly improving the depth and breadth of customer service. Depth means that under the same business scenario, with the development of business and the growth of volume, more enterprises are helped to bear the extreme pressure brought by business. Breadth is aimed at helping more enterprises respond quickly and adapt to new market opportunities brought about by changes through technological innovation with the emergence of new technology forms and business scenarios.

OceanBase is committed to exporting the most concentrated, classic and universal methodology accumulated by Ant's own business for many years to a large number of enterprise customers, while achieving both depth and breadth, and truly helping customers achieve stable innovation.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.