Shulou(Shulou.com)06/01 Report--

On July 19-20, 2019, the DAMS China data Intelligent Management Summit with the theme of "planning strategies for the future" was successfully held in Qingpu District, Shanghai. At the DAMS Summit, Giant Sequoia Database brought you the theme of "distributed database design and practice under cloud architecture".

The Evolution of Database Architecture under Micro Service

In the transformation of application development from traditional architecture to distributed architecture, the first thing to face is the application framework. Today's micro-service framework is very mature, and its representative architecture often includes four layers: protocol processing, service assembly, atomic services, and underlying persistence. The business logic is disassembled from the traditional single middleware into a large number of micro-service modules, each of which is composed of a series of completely equivalent containers, which can simply expand the throughput capacity of the service by adding containers.

But the split of micro-services means that each service has its own independent execution logic and storage. From the point of view of database, the split of micro-service system poses a great challenge to database storage. If each microservice still stores data in a traditional single point database, its storage and processing capacity cannot provide the same scalability with the increase in the number of microservice containers. In this case, the database will become the biggest bottleneck of performance and scalability in the micro-service architecture.

If each microservice is stored in a separate database, the data architecture of the entire enterprise IT will become fragmented. The number of databases has been split into tens of thousands of databases from the past hundreds, and the management costs and database procurement costs of the entire operation and maintenance team are facing a geometric increase.

Therefore, the goal of distributed database is not only as a single substitute for traditional Oracle or DB2, but also to store the data that cannot be stored in one database on multiple physical machines. In the actual environment, most banks have relatively perfect data life cycle management strategies, and generally do not accumulate a large amount of historical data in the production environment, so the amount of data is generally not the most important reason for the use of distributed databases.

Technical characteristics of New Generation distributed Database

In order to support the future IT micro service framework, the introduction of distributed transactional database needs to be evaluated from two dimensions: the compatibility of traditional technology and the foresight of new technology.

"when dealing with the micro-service application framework, the advantage of the native distributed database architecture lies in two points. The first is to solve the elastic expansion of data, so that each micro-service is not limited by the underlying data storage. Second, to solve the problem of serious data fragmentation in micro-service application architecture."

The support of ACID and the support of SQL integrity are two key indicators to evaluate whether a new distributed database is compatible with traditional database technology.

Support for ACID

From the security point of view, whether using new technology or traditional technology, data is good and not lost is a necessary basis for all databases. In the distributed database industry, some products designed for Internet technology aim at Partition Tolerance and Availability, which cannot guarantee the correctness of data in security consistency (Consistence), so it is difficult to be widely used in financial business. Therefore, the new distributed database concerned by banks must first ensure the security and consistency of data, in which distributed transactions, distributed locks and the support of four isolation levels are the key technical points in this index.

SQL integrity support

SQL integrity refers to the development friendliness of new distributed databases and traditional relational databases. The more mature the distributed database is, the more compatible its SQL syntax is with the traditional relational database, and the more transparent its data segmentation is to the application program. Nowadays, most distributed database technologies claim to support MySQL syntax, and mainstream new applications also take MySQL as their default database option. Therefore, the strength of the support for MySQL syntax protocols is the key to evaluate the SQL integrity support of distributed databases.

The new technology forward-looking refers to whether the distributed database is consistent with the future development method and IT architecture.

Distributed and resilient scalability

As a data service resource pool, distributed database must be flexibly expandable in order to continuously increase the type and number of micro-services in the upper layer. At the same time, for each micro-service, whether the data is stored in one physical device or multiple physical devices, the application code must be completely transparent.

Multi-mode engine

To serve the upper micro-services from different developers, different business scenarios and different data types, distributed database must support a variety of SQL protocols and computing engines. From the perspective of storage engine, both structured and semi-structured data may be used in applications at the same time. Therefore, the new generation of distributed database needs to support Multi-Model engine from access interface to storage structure.

HTAP (Hybrid Transactional/Analytical Processing)

HTAP is the ability to analyze and process mixed transactions. In the traditional bank IT architecture, the online transaction and statistical analysis system often uses different technologies and physical equipment, and migrates the online transaction data to the analysis system through the ETL executed regularly. As a data service resource pool, the same data may be shared and accessed by different types of micro-services. When some online transactions and audit services run at the same time for the same data, it is necessary to ensure that the requests are executed in a completely isolated physical environment, so that the transaction analysis business does not interfere.

Generally speaking, the trend of distributed database technology needs to be judged from the two dimensions of traditional technology compatibility and new technology foresight, in which ACID data security and SQL integrity are important indicators of traditional technology compatibility, while elastic scalability, multi-mode engine and HTAP are several important measures of new technology foresight.

Technical advantages of SequoiaDB Giant Sequoia Database

Cloud management under micro-service architecture

Under the trend of micro-service application development and cloud platform in the future, applications are no longer built with "chimney-like" middleware plus database mode, but a complex mesh model built by thousands or even tens of thousands of micro-service programs.

Aiming at the micro-service architecture, the new generation of distributed database will provide instantiated deployment of the database to help better interface between the database and upper-level micro-services, and the database will also provide cloud management to ensure that large and complex micro-services can achieve finer-grained management.

For micro-service application architecture, users can create hundreds of different relational database instances in the same cluster. The access and usage of the database instance is 100% compatible with that of the traditional relational database, and the data used at the underlying layer is logically independent. Each instance has its own independent rights management, data control, and can even choose to deploy in a separate hardware environment or shared devices.

The SequoiaDB Giant Sequoia database supports instantiation of database services. From the perspective of application developers and DBA, the relational database instances provided by SequoiaDB Giant Sequoia database are fully compatible with traditional MySQL, PostgreSQL, and SparkSQL.

Storage resource pooling and elastic expansion

With the increasing demand for massive data management, the expansibility of database storage has become an important core capability to support the whole data platform. SequoiaDB uses sharding technology to provide a flexible horizontal expansion mechanism for the system, which can adjust storage according to application requirements, and realizes multi-mode data management of storage engine, supporting full coverage and unified management of unstructured, structured and semi-structured data, making database platform scenarios more diverse and meeting the requirements of unified management and operation and maintenance of diversified business data under cloud data architecture.

The sharding process of SequoiaDB is completely transparent to the application. This mechanism solves the problem that the hardware resources of a single server (such as memory, CPU, disk I _ par O) are limited, and does not increase the complexity of application development. Through the native distributed architecture, Giant Sequoia database can easily achieve PB-level data management. Currently, the production environment supports a maximum of more than 1500 node clusters to achieve full data lifecycle management.

At the same time, in a large cluster, SequoiaDB provides multi-dimensional, multi-level logical and physical isolation capabilities. In a typical data resource pool type basic data service platform (DBaas,DataBase As A Service), SequoiaDB sequoia database can serve hundreds of applications with different SLA service levels, priorities, business characteristics, and data volume at the same time, and ensure the logical and physical isolation of data between applications.

Distributed transaction and transaction

From the security point of view, whether using new technology or traditional technology, data is good and not lost is a necessary basis for all databases. In the distributed database industry, some products designed for Internet technology aim at Partition Tolerance and Availability, which can not guarantee the correctness of data in Consistence, so it is difficult to be widely used in banking business. Therefore, the new distributed database concerned by large enterprise users must first ensure the security and consistency of the data, in which distributed transactions, distributed locks and the support of four isolation levels are the key technical points in this index.

Under cloud and micro-service architecture and SequoiaDB distributed architecture, it provides complete ACID support, complete transaction support and consistency guarantee, fully ensures data security and meets the stringent requirements of core trading business.

Compatibility of traditional applications

As a new generation of distributed database, its architecture and functional features need to ensure that it embraces micro-services and cloud computing framework on the basis of full compatibility with traditional databases. Therefore, for distributed transactions and ACID, distributed database must be fully compatible with traditional technologies. At the same time, when facing micro-service application development and cloud computing infrastructure, the new generation of distributed database must support a series of functions, such as flexible expansion, resource isolation, multi-tenancy, configurable consistency, multi-mode (supporting all kinds of SQL protocols), configurable disaster recovery strategy within the cluster, and so on.

Among them, the support of SQL is the most important technical feature of database products. SequoiaDB provides native and complete SQL support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.