Shulou(Shulou.com)06/01 Report--

Introduction: when WeBank was founded in 2014, she was very forward-looking and established the infrastructure of distributed architecture. At that time, Tencent had a financial-grade distributed database product, TDSQL, and its business scenario and reliability requirements for the database were very similar to the banking scenario. WeBank and Tencent TDSQL team work together to build TDSQL into a financial-level distributed database product suitable for bank core scenarios, and use TDSQL in WeBank's core system database. This paper is a summary of the whole process of practice.

First, background introduction

When WeBank was founded in 2014, it was very forward-looking to establish the direction of WeBank's IT infrastructure: abandon the traditional centralized architecture model based on commercial IT products and take the Internet model of distributed architecture. As we all know, the traditional bank IT architecture system is very dependent on the traditional commercial database, commercial storage and large and medium-sized server equipment, and it also needs huge IT costs to maintain and upgrade every year. At the same time, this centralized architecture is not convenient for efficient horizontal expansion. From past experience, in addition to a few traditional commercial databases such as oracle, there were not many database products that could meet the financial-grade banking scenario.

At that time, Tencent had a financial-grade distributed database product, TDSQL, which mainly carried Tencent's internal billing and payment business. Its business scenario and reliability requirements for the database were very similar to the bank scenario, and it was also verified by Tencent's massive billing business scenario.

After many rounds of evaluation and testing, WeBank's infrastructure team finally decided to work with Tencent TDSQL team to build TDSQL into a financial-level distributed database product suitable for bank core scenarios, and to use TDSQL in WeBank's core system database.

2. Why TDSQL?

Why did you choose TDSQL as WeBank's core database? This section details the TDSQL architecture and the core features of TDSQL to see how TDSQL meets the database requirements of financial-level scenarios.

Introduction to 1.TDSQL Architecture

TDSQL is a financial-level distributed database cluster scheme based on MySQL/Mariadb community version. At the kernel level, TDSQL makes system-level optimization in the replication module for the kernels of MySQL community version and Mariadb community version, so that it has the feature of strong consistent synchronization of master and standby data, which greatly improves data security. At the same time, compared with the native semi-synchronous replication mechanism, the performance of TDSQL strong consistent replication is also greatly improved. TDSQL integrates TDSQL Agent, TDSQL SQLEngineSQLEngine, TDSQL Scheduler and other modules, and realizes a series of functions such as read-write separation, AutoSharding, automatic active / standby strong consistency switching, automatic fault repair, real-time monitoring, real-time cold backup and so on. The TDSQL architecture model is shown in the following two figures.

Figure TDSQL architecture model and SET model

We can understand the architecture of TDSQL from both horizontal and vertical dimensions.

Horizontal is the request processing path of TDSQL: the request is sent through APP, passed through the load balancer module, and forwarded to the TDSQL SQLEngine cluster; after TDSQL SQLEngine receives the request, the request is parsed, and then forwarded to the database instance node in the set unit (write request to master, read request can be sent to master or slave); after the database instance processes the request, it is returned to TDSQL SQLEngine,TDSQL SQLEngine and then sent back to app through the load balancer module.

Vertically is the management path of the TDSQL cluster: a snap-in of TDSQL is called a set, and a TDSQL Agent module is deployed on each database instance of each set unit. The Agent module collects all the monitoring information of the database instance (including node master and standby role information / node survival status / request volume / TPS/CPU load / IO load / slow query / number of connections / capacity utilization, etc.) and reports it to the ZooKeeper cluster. ZooKeeper is equivalent to the entire TDSQL cluster metadata storage and management center, saving all the metadata information of the cluster. The TDSQL Scheduler module monitors the reported information stored by ZooKeeper and initiates different scheduling tasks according to the status of the cluster, which is equivalent to the brain of the TDSQL cluster and is responsible for the management and scheduling of the whole cluster.

2. TDSQL noshard and shard mode

TDSQL provides two usage modes, noshard and shard, as shown in the following figure.

Figure TDSQL noshard and shard mode

The so-called noshard mode is a single-instance mode, which is not automatically divided into databases and tables, and is fully compatible with MySQL in syntax and function. The disadvantage is that it only supports vertical expansion, which is limited by the performance and capacity limit of single-instance servers, and cannot be scaled horizontally.

Shard mode is AutoSharding mode. Through the TDSQL SQLEngine module, the Sharding and distributed transaction functions of the database are realized, and the underlying data is scattered on multiple database instances, and the application layer is still a unified single database view. Shard mode can achieve horizontal expansion of capacity and performance, and supports distributed transactions and various related operations through two-phase XA, but stored procedures are not supported yet. At the same time, business-specific shard key is required when building tables, which will be intrusive to some business development.

WeBank fully considered whether to use shard version of pure distributed database or distributed from the perspective of application layer when he was doing the system architecture at that time. Through a great deal of investigation and analysis, he finally felt that using application to do distributed was the most controllable, secure, flexible and scalable mode. Thus, he designed a distributed scalable architecture based on DCN, which was split horizontally at the application layer. The database adopts TDSQL noshard mode, which ensures the simplicity of the database architecture and the compatibility of the business layer, which will be discussed in detail later.

3. Strong consistent handover and second recovery between active and standby

TDSQL achieves strong consistent replication of multiple replicas in a real sense by customizing the source code of the MySQL kernel. Through the active and standby deployment mode, it can achieve RPO=0, that is, zero data loss, which is essential and the most basic requirement for financial scenarios. At the same time, based on the modules such as TDSQL Agent and Scheduler, the automatic strong consistent handover between master and standby is also realized. The whole process of active and standby switching can be completed in 30 seconds, and the second recovery of fault RTO is realized.

4. Watch node mode

The TDSQL slave node provides two roles, one is the follower node and the other is the watch node. Both Fllower node and watch node synchronize data from master node in real time, but watch node does not participate in active / standby election and active / standby switching, but only synchronizes data as an observer. The roles of Follower nodes and watch nodes can be adjusted online in real time.

5. Automatic monitoring and operation and maintenance

TDSQL provides Red Rabbit management platform system to support the visualization, automatic monitoring and operation and maintenance functions of the whole TDSQL cluster. As shown in the following figure, it is the running interface of the TDSQL Red Rabbit management platform.

Figure TDSQL Red Rabbit Management platform

Through the TDSQL Red Rabbit management platform, a series of functions such as monitoring data collection and display, alarm and policy configuration, daily operation and maintenance (master / standby switching, node replacement, configuration change, etc.), database backup and recovery, slow query analysis, performance analysis and so on can be realized, which greatly improves the efficiency and accuracy of operation and maintenance.

Based on the above architecture and characteristics of TDSQL, we believe that TDSQL meets the requirements of high availability, high reliability, operation and maintenance of database in financial business scenarios. At the same time, the software and hardware platform based on MySQL and X86 can also greatly reduce the cost of IT at the database level, thus greatly reducing the cost per household, which is very suitable for the new generation of banking architecture in the Internet era.

Third, distributed extension architecture based on DCN

As mentioned earlier, in order to achieve the horizontal expansion of the business scale, WeBank designed a distributed scalable architecture based on DCN, which not only realizes the scalability, but also ensures the simplicity of the database level architecture.

DCN, that is, Data Center Node (data center node), is a logical region concept, and DCN is a self-contained unit, including the complete application layer, access layer and database. Can be understood as a DCN, that is, an online virtual branch of WeBank, this virtual branch only carries a part of WeBank's business customers. Through certain routing rules, such as account number segmentation, different customers are divided into different DCN. Once the number of customers carried by a DCN reaches the prescribed limit, no new customers will be added to the DCN. At this time, through the deployment of a new DCN, to achieve horizontal capacity expansion, to ensure the sustained and rapid development of the business.

If different customers are stored in different DCN, it is necessary to have a system to keep the global routing information and record which DCN a customer is in. This system is GNS (Global Name Service). The application module will first request GNS, get the corresponding customer's DCN information, and then request the corresponding DCN. GNS uses redis cache to ensure high query QPS performance, while TDSQL is used for persistent storage to ensure data security.

RMB (Reliable Message Bug), reliable message bus, is another core module of DCN architecture, which is mainly responsible for efficient, accurate and fast message communication between various business systems. The overall architecture of DCN is shown in the following figure.

Figure DCN architecture model

Fourth, WeBank IDC architecture

With the infrastructure model based on DCN, the next step is to build the basic physical environment. After more than four years of development, WeBank has now developed into a framework of two places and six centers, as shown in the following figure.

Picture WeBank IDC architecture

Among them, two places are located in Shenzhen and Shanghai, Shenzhen as the production center, there are five IDC computer rooms in the same city in Shenzhen, and Shanghai has one IDC computer room as a cross-city disaster recovery. There are 5 IDC in the same city in Shenzhen, which are interconnected by multiple direct connect lines to ensure extremely high network quality and bandwidth. At the same time, the distance between any two IDC is controlled at 10-50 km to ensure that the network ping delay between computer rooms is controlled at about 2ms. This is very important and is a prerequisite for implementing TDSQL inter-city cross-IDC deployment.

5. Multi-activity of the same city application based on TDSQL

Based on the above DCN architecture and IDC architecture, we designed the deployment architecture of TDSQL database in WeBank. This is shown in the following figure.

Picture WeBank's architecture of multiple activities in the same city based on TDSQL

We use the 3x2 noshard deployment model of 3 replicas in the same city + 2 replicas across cities. The 3 replicas in the same city are 1 master and 2 slaves. In the 3 IDC deployed in the same city, strong consistent synchronization of TDSQL is used between the replicas to ensure the RPO=0,RTO recovery between 3 IDC in the same city. The cross-city 2 replicas are replicated asynchronously through a slave in the same city to achieve cross-city data disaster recovery. Based on the above architecture, we can achieve multi-application activity in the same city, that is, online business traffic can be accessed from three IDC at the same time. If any IDC failure is not available, zero data loss can be guaranteed, and database service can be restored within seconds.

In the same IDC, the ping latency between servers is usually less than 0.1ms, while the ping latency between inter-city and cross-IDC servers will greatly increase. Will that affect the performance of TDSQL master / slave strong synchronization? In addition, can the stability of the network between IDC be guaranteed?

We take the following measures to eliminate or avoid this problem.

First of all, at the infrastructure level, we will ensure that the distance between the three IDC in the same city is controlled at about 10-50 km, and the network delay is controlled at about 2ms; at the same time, we build multiple direct connect lines between the IDC to ensure the quality and stability of network transmission; secondly, TDSQL makes a large number of kernel-level optimizations for this cross-IDC strong synchronization scenario, such as queue asynchronization and concurrent replication. The benchmark test shows that the impact of strong synchronization across IDC on the performance of online OLTP is only about 10%.

From the perspective of our actual production and operation, this inter-city cross-IDC deployment model is completely acceptable for the performance impact of online OLTP business, but for centralized batch scenarios, because of the cumulative effect, it may eventually have a greater impact on the completion of the batch. If a batch APP needs to access the database across IDC, the network latency of each access to the database will be accumulated and magnified throughout the batch period, which will seriously affect the batch running efficiency. In order to solve this problem, we make use of the watch node mechanism of TDSQL. For the TDSQL SET participating in batch running, we deploy an additional WATCH node with the same IDC as the master node on the basis of the original one master and two backups, while ensuring that the batch APP and the master node are deployed in the same APP. This is shown in the following figure.

Figure TDSQL deployment model with WATCH node

The WATCH node and the master node are deployed in the same IDC, and the data is synchronized asynchronously from the master node. Because the WATCH node is synchronous asynchronously, the binlog of the master node will ensure synchronization to the transactions of the other two standby nodes across the IDC. In this way, even if the IDC of the master node is completely down, the integrity of the data can still be guaranteed and the disaster recovery characteristics of IDC will not be affected. When the master node fails, the scheduler module compares the data consistency between the watch node and the other two strongly synchronous slave machines. If it is found that the data of the watch node is as new as the other two idc data (this is normal, because the same IDC is generally faster than cross-IDC), the watch node will be promoted to the host first. This ensures that the batch APP and the database master node are in the same IDC as far as possible, and the delay effect caused by cross-IDC access is avoided.

Through the above deployment architecture, we have achieved cross-city cross-IDC level high availability and multi-city cross-IDC application activity, which has greatly improved the overall availability and reliability of WeBank infrastructure.

VI. TDSQL cluster size

Since the establishment of WeBank for more than 4 years, the business has developed rapidly. At present, the number of effective customers has exceeded 100 million. WeiLiDai and micro-business loans have also become star products in the industry. In the process of the rapid growth of the business scale, the scale of our database is also growing. At present, the number of TDSQL SET of WeBank has reached 350 + (production + disaster recovery), the number of database instances has reached 1700 million, and the overall data scale has reached PB level, carrying hundreds of core systems of WeBank. In previous business peaks, the highest daily financial transaction volume reached 360 million +, and the highest TPS reached 100000 +, as shown in the following figure.

Figure WeBank TDSQL business scale

In the past four years of operation, TDSQL has never experienced any major system failure or data security problems. At the same time, the software and hardware architecture of X86 based on TDSQL has helped WeBank greatly reduce the per capita cost of IT and greatly enhance WeBank's competitiveness in the industry. WeBank has proved through practice that TDSQL, as the core database of financial level, is fully competent.

7. The present situation and future development of WeBank database

At present, TDSQL carries more than 99% of WeBank's online database business. At the same time, our bank also uses redis as a cache to solve hot scenarios such as second kill and panic buying. In addition, there are a small number of MongoDB to meet the storage needs of document classes. At the same time, since last year, our bank has also tried to introduce NEWSQL database TiDB to solve a small number of business scenarios that are unable to split DCN, but at the same time have a large demand for single-database storage capacity or throughput. On the whole, our current database mainly includes TDSQL,TIDB and Redis/MongoDB,TDSQL, which mainly carry the core system business. TIDB is used as a supplement to solve the non-online business requirements that a single database requires super capacity or super throughput, while Reids and MongoDB mainly provide cache and document storage. Of course, we will not stop here. WeBank database team and Tencent Cloud TDSQL team will have more in-depth cooperation in the future. For example, the TDSQL Intelligent OPS-Bian Que project, which we worked with Tencent Cloud TDSQL team, has been launched in WeBank gray level, and can analyze the running status and performance of TDSQL in real time, which is a sharp tool to improve the efficiency of operation and maintenance. We are also working with the TDSQL R & D team to investigate and evaluate MySQL version 8.0 and the high availability features of MySQL based on MGR. In the future, we may try to integrate MySQL 8.0 and MGR into the TDSQL system and try it in the bank's core system.

A brief introduction to the author

Hu Panpan, head of WeBank database platform. He graduated from Huazhong University of Science and Technology, joined Tencent as a senior engineer, engaged in R & D and operation related to distributed storage and cloud database, and joined WeBank in 2014. responsible for the design, planning and operation management of WeBank's database platform.

Huang Dezhi, WeBank database platform senior DBA. In 2009, he joined Ping an Technology and served as a senior database development engineer and a senior operation and maintenance engineer. In 2016, he joined WeBank as a senior DBA, responsible for TDSQL related operation and maintenance work.

Recommended reading

AI parameter adjustment surpasses database experts for the first time

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.