Shulou(Shulou.com)06/01 Report--

This article is reproduced from "Electronic Finance"

Original link: https://mp.weixin.qq.com/s/WGG91Rv9QTBHPsNVPG8Z5g

With the rapid development of mobile Internet, distributed architecture is widely used in the field of Internet IT technology and has accumulated a lot of practical experience. Under the environment of the rapid development of Internet finance and the marketization of interest rates, it has become an urgent need for the domestic financial industry to build a distributed architecture application system that can support a large number of customers, flexible expansion, efficient and flexible.

The general trend of distributed database application

The construction of our inclusive financial platform aims to "make full use of financial science and technology, optimize credit processes and customer evaluation models, reduce financing costs for enterprises, and alleviate the financing difficulties and expensive problems of private enterprises, small and micro enterprises." enhance the real economic capacity of financial services.

Inclusive financial service is a typical Internet application, which is different from the traditional credit system in that it has the ability to access Internet scenarios. If the centralized technical architecture is used, there are the following potential problems in dealing with the Internet application scenarios and total cost of ownership of a large number of customers:

Centralized architecture generally lacks the ability to scale flexibly. With the growth of transaction volume and data volume, the overall throughput of the system will encounter hardware or technical bottlenecks. Especially when supporting the business related to Internet customers, it can not effectively deal with the instantaneous outbreak of high concurrency transactions, which restricts customer acquisition and large-scale business marketing.

The centralized architecture adopts single application design. The smallest unit of software development and operation management is application, and the management is rough, so it is easy to "pull one trigger and affect the whole body". The development process of application is not easy to practice the concept of lightweight agile development, and the system is prone to single point of failure in the process of running. it is difficult to isolate faults effectively.

The infrastructure of centralized architecture systems usually uses high-end servers and storage devices, as well as traditional relational databases. The purchase cost of hardware and software is high, the development and operation and maintenance mainly rely on the service manufacturer, the service cost is high, and can not achieve complete self-control.

The technology system is closed, and the development of technology is highly dependent on manufacturers, especially on foreign manufacturers. The IT team of commercial banks lacks independent and controllable ability, and there are information security risks to a certain extent.

When planning the construction of a new generation of inclusive financial platform, our agency makes an in-depth analysis and reflection on the above issues from a strategic perspective, based on long-term development planning, and looks at the research and development of the distributed system from both horizontal and vertical angles, that is, focusing on distributed database in terms of data, and introducing micro-service architecture at the functional level, so as to truly realize the underlying system of financial services with omni-directional distributed framework. At the same time, in response to the national requirements for independent control of technology, the domestic distributed database is selected to effectively control the cost of IT and realize the self-control of technology.

Application practice of domestic distributed Database

The inclusive financial platform is different from the traditional business system, it has the ability to access the Internet scene, and the way of getting guests has changed. In order to meet the explosive growth of customer applications for loans, there have been changes in the examination and approval methods and processes, and there are higher requirements for enterprise credit information and intelligent risk control. With the intensification of competition in subdivided areas, the application of the platform needs to be constantly adjusted and iterated at any time.

After careful and strict selection, we finally adopted the domestic distributed database-SequoiaDB Giant Sequoia Database as the underlying distributed database platform.

The platform application uses the distributed database computing-storage layer separation technology to decouple the protocol parsing, computing and other modules from the underlying storage, the storage layer is elastically expanded through multi-dimensional partitions, and the computing layer is designed stateless and deployed independently. by dynamically increasing the database instance to linearly improve the computing power, it can effectively deal with the instantaneous outbreak of high concurrent massive transactions. The distributed database platform is fully compatible with MySQL, PosgreSQL and SparkSQL to provide high compatibility for applications.

The platform application development is based on the Spring Cloud micro-service framework, and the business logic is disassembled into many micro-service components from the traditional single middleware. The distributed database instance service layer provides a standard database computing engine that is selectable, adaptable and dynamically scalable, which can be chosen flexibly according to the application scenario. The application development uses the database distributed computing engine, facing the online transaction business scenario, using MySQL-compatible computing examples, and batch processing business scenarios using SparkSQL-compatible computing examples. The example realizes the parallel processing and multiple utilization of the data (one data, multiple uses) by accessing different data copies of the distributed database.

The following are several technical points of distributed database for online business applications:

1. Application Design under distributed Architecture

The core mechanism of distributed database is to split massive data into one or more partitions through a certain algorithm. Each data partition can use different physical servers and connect the servers together through network devices to form a logically unified database service. Therefore, how to divide the data evenly and how to retrieve the data quickly after segmentation are the priority contents that must be guaranteed by the distributed strategy.

Distributed database segmentation rules include:

(1) Hash partition: hash partition refers to the rule that determines the physical storage partition of the data after hashing the values of the specified partition field in the process of data writing.

(2) range partition: range partition refers to the rule that in the process of data writing, the range of the specified partition field is judged according to the value of the specified partition field, and the physical location of the data partition is determined according to the data partition corresponding to the data range.

(3) Multi-dimensional mixed partition: multi-dimensional mixed partition is the combination of hash partition and range partition. In a multi-dimensional hybrid partition, users can specify two different fields when creating a table, first determine the range according to the first field, and then hash the location of the second field in a series of physical servers corresponding to the range.

In the new generation inclusive financial service platform project of our society, the design of database table structure is targeted to define partition fields and rules for data features. In the process of practice, it greatly improves the efficiency and concurrency of the application, and avoids the problem that the database has become a bottleneck and hot spot in the traditional architecture.

two。 Data consistency guarantee under distributed Micro-Service Architecture

When using the distributed database system to achieve the underlying business support, it will bring many characteristics that can not be compared with the traditional centralized database, such as distributed fault tolerance, flexible scalability and so on. these are all important optimization factors for banks and other financial industries. The distributed database system can have these capabilities mainly because of its distributed storage characteristics, that is, each node saves part of the complete database and deploys its copies on several other nodes. Then, in order to realize such a distributed database system and make it give full play to its due ability, a very important part lies in its consistency strategy.

In the course of practice, we have focused on studying and solving the recognized problems of end-to-end data consistency and data security in the distributed environment:

(1) using flexible transactions to ensure data consistency between microservices

Through the platform research on the application strategy of distributed database in the banking field, we deeply understand the distributed underlying technology, and use flexible transaction TCC to achieve data consistency between micro-services. The application development of Pratt & Whitney financial platform is based on the Spring Cloud micro-service framework. The application micro-service development specification makes it clear that each group of micro-services must provide try-confirm-cancel interface, follow the RESTful TCC solution proposed for the distributed transaction of micro-services, and combine Spring Cloud Netflix and distributed flexible transactions to achieve data consistency between micro-services.

(2) within the micro-service, automatic transaction consistency guarantee is realized through distributed database technology.

Combined with the characteristics of the business data of Pratt & Whitney financial system, the multi-dimensional partition strategy of data is used reasonably, and the distributed database technology is used to realize the automatic transaction consistency guarantee for abnormal fault scenarios on the basis of two-phase commit (2PC).

In order to ensure the consistency of data, it is necessary to maintain a globally unique ID to distinguish concurrent transactions and identify the data generated or changed. The implementation of such a globally unique ID usually combines the time of transaction initiation and depends on timestamps plus other identification bits. The domestic distributed data system specially designs a set of fully distributed logical clock mechanism, which avoids the step of forcing all participating nodes to obtain unique ID from GTM, and at the same time meets the requirements of distributed storage and processing.

3. Distributed database deployment across data centers in the same city

Based on the perfect multi-copy data synchronization mechanism of the distributed database, the deployment of dual centers across the same city is realized, the disaster recovery and high availability of data are supported, and the applications of the new generation inclusive financial platform run across data centers. When there is a disaster in the main center, zero data loss, distributed database automatic disaster recovery switch, completely transparent to the application system. The following figure is a schematic diagram of the dual active deployment of the distributed database application of the Provincial Associated Press inclusive Finance platform:

Project innovation achievements

1. Distributed Architecture Transformation and Financial Localization

Distributed database provides flexible, flexible, convenient and integrated data service capabilities to cope with the growth of instantaneous explosive business. Combined with micro-service architecture, it lays a good foundation for agile iterative development of applications. Innovative, differentiated, cross-border financial services and products are launched quickly.

Using the domestic distributed database giant sequoia database, the underlying code can be completely controlled independently, the foreign database products can be replaced, and the binding of core technology can be avoided. The overall improvement is achieved from two aspects of software and hardware, performance and efficiency, tactical optimization of short-term application and strategic optimization of long-term development. In addition, considering the remote disaster preparedness, through the multi-activity deployment and practice of the combination of distributed data and applications, it has explored a feasible way for the construction of multi-live data centers for rural credit cooperatives and other financial institutions in the future.

two。 End-to-end data consistency in distributed environment

In the course of practice, we focus on studying and solving the problems of end-to-end data consistency and data security in the industry's recognized distributed environment: (1) using flexible transactions to ensure data consistency between micro-services; (2) within micro-services, automatic transaction consistency guarantee through distributed database technology (3) using distributed database multi-activity system to solve the problem of data consistency and high availability across data centers.

Under the framework of micro-services in the virtual cloud environment, we have completed the multi-level data consistency of the distributed system, thus achieving excellent characteristics such as efficient elasticity and easy expansion, ultimate consistency of transaction processing, micro-service-level data protection and multi-live remote disaster recovery, providing a strong driving force and development potential for the further upgrading and reform of banks and other financial systems under the background of financial technology.

3. Maximize social and economic benefits

We have conducted in-depth research on the Internet financing industry and refined the commonness of Internet financing business. based on the flexible multi-level hybrid platform system, we have launched a strong and stable inclusive financial platform that integrates customer accounts, product innovation, full-cycle accounting, automatic business examination and approval, and intelligent risk control. The platform provides customers with differentiated configuration of financial services, and completes the Internet cross-border product integration, service interaction, product accounting and other functions. It not only provides a comprehensive package of inclusive finance and Internet financing services, but also lays a solid foundation for the rapid development of corporate Internet financial business.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.