Shulou(Shulou.com)06/03 Report--

The following brings you the technical requirements and architecture evolution of cloud database, hoping to give you some help in practical application. Cloud database involves a lot of things, there are not many theories, and there are many books on the Internet. Today, we will use the accumulated experience in the industry to make an answer.

Today, large enterprises, such as financial enterprises and banks, under the requirements of the next generation of micro-service architecture transformation, need basic software and data platform to achieve native cloud to meet the needs of micro-service architecture.

Micro-service, that is, a service-oriented, loosely coupled architecture with specific boundaries.

The main features include that each micro-service is an independent autonomous system, which can run independently without external components; only exposing the interface to the application, users can flexibly adjust the use of each micro-service; the business granularity is small enough.

In the process of "cloud" of enterprise architecture, the cloud of database is the most important and difficult part. Database cloud platform (dbPaaS) is a kind of database management system (DBMS) or storage management system that supports elastic expansion, multi-tenancy, self-management, and can run on the cloud service provider's infrastructure (IaaS).

According to the Gartner report, the market share of database cloud platforms will double in the next five years, while 70% of users will start using dbPaaS database cloud platforms. Therefore, in order to meet the needs of various applications for database cloud platform, and to reduce the complexity of operation and maintenance of a large number of different types of data storage products in private cloud deployment, the evolution of database architecture will be one of the main directions of database transformation in the next decade.

Technical requirements of Cloud Database

In the process of "cloud" of business and application, the importance of cloud database in cloud transformation is self-evident because of its important position in the overall architecture. The core requirements of cloud database are as follows:

elastic expansion capability: database capacity needs to be expanded flexibly according to business to meet the capacity needs of different businesses.

flexible deployment and on demand capability: in addition to database storage, other database functions also need to be flexibly deployed according to the needs of the application.

data reliability and service sustainability: reliable and secure data, full-time online is a necessary requirement for all businesses

Separation of computing and storage: flexible allocation of computing and storage resources, so that you can choose not only multiple computing methods but also multiple storage methods at the same time to meet more business needs.

multi-schema storage capacity: structured, unstructured, semi-structured, graph and other types of data storage

self-management capabilities: provide zero downtime maintenance, continuous integration, and rolling upgrade capabilities to improve developer efficiency

self-monitoring and problem repair capabilities: fault monitoring and problem repair to reduce operation and maintenance costs

Whether meets specific application scenarios: pluggable components or tools for specific scenarios

regulation and security: meet the regulatory requirements and ensure the security of data.

Cloud database needs to meet these technical requirements, in addition to the specific improvement in function, but also needs to be upgraded and "evolved" in the overall architecture.

The direction of cloud database architecture

Cloud database architecture is the key point of whether it can carry the "cloud" of application architecture. With the development of technology and business, the architecture of cloud database has appeared several main development directions:

in the dbPaaS platform, computing-storage layer separation will become the mainstream technology direction. By decoupling protocol parsing, computing and other modules from the underlying storage, the database cloud platform splits the storage layer to achieve the elastic horizontal expansion of the storage. at the same time, the stateless design of the computing layer allows the computing layer to linearly improve computing power by increasing the number of nodes, which has reached the elastic level expansion of the entire database cloud platform.

The composition of multi-mold base is the mainstream trend. The architecture of Multi-model can support multiple storage methods in one database platform, which greatly reduces the cost of operation and development. Storage engines such as relational, OO, and even XML have long been provided in traditional databases such as IBM, Oracle, etc. The new generation database provides many types of data storage engines, such as NewSQL, JSON, graph, object storage and so on.

The cloud database platform will provide a variety of mixed-mode data services-relational and non-relational. This mode enables users to combine the characteristics of different data storage types in the same platform to provide a hybrid data storage solution for the new generation of IT application systems.

is more in line with the requirements of micro-service business architecture. Micro-service requires loose coupling and independent expansion between service modules as far as possible. Therefore, for the database, it will also focus on different configurations for different businesses, whether it is the traditional "read-write separation" or the popular HTAP around this requirement.

In view of these main development directions, we will discuss several important technical features of cloud database in detail.

1) Storage-SQL separation

According to the demand and architecture direction of cloud database, a new database architecture is gradually becoming mainstream, that is, the "storage-SQL separation" architecture of the database.

Storage-SQL separation architecture, that is, the storage engine and SQL engine of the database are loosely coupled and work independently of each other. Usually this architecture is divided into three parts: storage, SQL and metadata.

storage layer: the storage engine of the database, which is responsible for the storage management of the data. At the same time, it includes routing and transaction control to ensure the ACID characteristics of the data. In addition, the storage layer should also have a series of functions, such as indexing, query condition filtering, sorting and so on.

SQL layer: the SQL layer is mainly responsible for handling SQL requests. The upper layer faces the application directly, distributes the application access requests to the storage layer, and accepts the data results returned by the storage layer.

metadata area: the metadata area is responsible for storing all metadata information for the entire database.

Typical cloud database architecture

For this architecture, in fact, the current architecture of the MySQL database is somewhat similar.

The SQL and storage separation architecture of MySQL database is more flexible in architecture, and its open source ecology also supports the full docking of different products, engines and tools. In the architecture of storage engine, the plug-in storage engine architecture separates query processing from other system tasks and data storage and extraction. This architecture can choose the appropriate storage engine according to the needs of the business and the actual needs.

The overall technical module architecture of MySQL database

As shown in the figure above, MySQL's storage engine can mount a variety of different products, each providing different technical features. It includes InnoDB, MyISAM and other architectures.

The architecture of storage and SQL separation is very popular in the database industry, and AWS's Aurora database also uses a similar architecture in SQL access. SequoiaDB 3.0 is currently compatible with MySQL, which mainly adopts the architecture of "SQL- storage separation".

SequoiaDB 3.0 MySQL compatible logical architecture

SequoiaDB 3.0 uses the native SQL parser of the MySQL database, which naturally supports the MySQL protocol and is 100% syntax compatible. In this architecture, the MySQL protocol parsing layer, as the role of SQL parsing and distribution, directly faces the application. Each MySQL service access node is an independent MySQL process that supports read and write operations. The data storage and management are completely realized by the distributed database engine of Giant Sequoia database. To put it simply, SequoiaDB 3.0, as the InnoDB replacement engine of MySQL, naturally supports all the syntax and functions of MySQL, while providing the ability to expand the database storage layer flexibly.

2) Multimode Multi-Model

More and more enterprises use cloud database interfacing applications with a variety of requirements. The traditional approach is to provide more than a dozen different database products in dbPaaS to meet various needs. With the increase of the system, the cost of overall maintenance and data consistency management is very high, which will affect the use of the entire system.

"Multi-mode" schematic diagram of cloud database

In order to achieve unified business data management and data fusion, the new database needs to have the ability of multi-mode (Multi-Model) data management and storage. Database multimode Multi-Model means that the same database supports multiple storage engines, which can meet the unified management needs of structured, semi-structured and unstructured data.

Generally speaking, structured data refers to the data storage structure of form type, and typical applications include traditional businesses such as bank core transactions; while semi-structured data is widely used in scenarios such as user portraits, Internet of things device log collection, application clickstream analysis, and unstructured data corresponds to a large amount of picture, video, and document processing business, which is growing rapidly with the development of financial technology.

Multi-mode data management capability enables financial-level databases to uniformly store and manage cross-departmental and cross-business data, achieve multi-business data fusion, and support diversified financial services.

In terms of architecture, the multimode Multi-model just mentioned is also aimed at the needs of the cloud database, which enables the database to use a set of data management system to support multiple data types, so it supports a variety of business models and greatly reduces the cost of use and operation and maintenance.

3) disaster preparedness and active activities

For applications, developers do not want to spend a lot of energy in the process of designing applications to consider the switching logic of applications with high availability of underlying data, disaster recovery and multi-active applications. Generally speaking, a mature dbPaaS layer should encapsulate a series of operations such as data multi-copy synchronization, disaster switching, high availability takeover and so on as far as possible, so as to be completely transparent to the application.

In traditional application development, developers use middleware containers to configure data sources, and the underlying layer uses F5 or other virtual IP addresses to encapsulate multiple data sources. However, in the process of cloud evolution, the underlying database changes from a single node to a distributed node. On the one hand, the upper application program hopes to reduce the dependence on sub-database and sub-table as much as possible, on the other hand, it is more desirable to achieve transparency and imperceptibility in the process of data node switching and even disaster takeover in the data center.

SequoiaDB 3.0 introduces a remote and multi-active architecture, and applications can access the local database in a read-write manner from any access node. In the process of data reading and writing, the giant sequoia database can effectively control the data consistency from the bottom, replicate the data written locally in multiple areas remotely, and ensure that the data read and written by multiple sites are completely consistent.

In addition, when a disaster occurs, the giant sequoia database provides a transparent data switching and takeover mechanism to the application, and dynamically adjusts the underlying data distribution topology logic, which can dynamically and effectively eliminate the nodes in the data center. so that other sites continue to provide data services without awareness.

Compared with the traditional high availability, multi-activity not only achieves a greater improvement in performance and security, but also can fully apply software and hardware equipment and reduce redundancy in multi-live data centers.

Advantages of cloud database architecture

Under the technology-driven requirements, the cloud database architecture has several major business values:

does not need sub-database and sub-table: previously, a kind of database distributed transformation direction is relational database to distributed architecture transformation, MySQL sub-database sub-table is one of the solutions. Today, the storage-SQL separation architecture, which has been natively implemented step by step in the data storage layer, avoids the complex and lengthy "database and table" scheme.

flexibly supports business requirements: both storage and SQL layers can achieve flexible adjustment of services and storage, and flexibly support business needs.

multi-storage engine compatibility: due to the separation of SQL and storage layer, the underlying storage engine can support multiple different engines and achieve simultaneous compatibility of multiple data engines while keeping the SQL interface unchanged.

is fully compatible with existing applications: because the SQL layer uses more existing standard SQL parsers, the original applications can be fully compatible on SQL without any investment in application transformation.

data is safely available: distributed storage and loosely coupled architecture, data has secure multiple copies, loose coupling greatly enhances the fault tolerance of the whole system. Compared with the traditional single-point architecture, it can well achieve the double-live or even more-active structure of data, and meet the regulatory and safety requirements of "two places, three centers" and "three places and five centers".

Cloud database application scenario

Driven by the new architecture, cloud database has begun to implement landing applications in many scenarios.

Traditional transaction service

In the traditional centralized transaction business, high-performance, high-throughput data storage and processing capabilities, ACID and security are all very important features. For example, in a typical banking business, in order to meet the online transaction volume during the peak period, the transactional database needs to process thousands of thousands of transactions per second in the database with hundreds of millions of records. At the same time, in order to meet the robustness and reliability of the production system, traditional transaction services have very clear requirements for the security, high availability and deployment capability of the underlying data storage.

Therefore, cloud database not only needs to gradually transfer the traditional transactional business to the cloud platform, but also needs to provide better support for users in terms of security and compliance regulation.

Historical data service

In recent years, with the continuous development of IT technology and big data, more and more enterprises retain data as their precious assets for a long time. This makes the historical data burden of some traditional applications become heavier and heavier, and finally the database is overloaded and the overall performance of the application is poor. On the other hand, with the increasing demand of big data, the data that has been archived needs to be online again to meet the requirements of online, real-time use, query and analysis, which requires the original huge offline data to be "online". These requirements make historical data management necessary.

For historical data services, due to the direct access to external applications, its robustness, reliability, configurable consistency policy, performance and concurrency are of great concern. At the same time, compared with traditional trading services, strong consistency and ACID are not the most concerned points. In view of the fact that some enterprises directly run some reports and self-help queries on the history service platform, the ability of HTAP is also a feature worth paying attention to.

Cloud database meets these requirements through distributed architecture in terms of scalability and performance, and manages the historical data well.

Real-time online service

At present, most of the production business systems and background data processing, analysis and query systems of most enterprises carry out data ETL by the way of Test1. Recently, with the rise of stream processing technology, more and more enterprises begin to build data bus based on stream processing technology to realize real-time data docking between different business processes. For example, the user asset view can make use of the flow processing technology to provide the excellent user experience of the full asset view query, while greatly reducing the query pressure on the background production system.

For real-time online services, the database level is most concerned about performance, throughput, reliability, and availability. Strong consistency, ACID, and HTAP are not the most important features.

The data diversification and performance of online services need more flexible and efficient support from cloud-based databases.

Image storage service

Many industries will produce a large number of paper vouchers in their business operations. under the requirements of information processing and supervision, these paper vouchers need to be scanned into image files and preserved for a long time. With the in-depth promotion of the concept of Internet technology and centralized operation center, a large number of industries generally need to build a unified image management platform.

For a typical image platform, the overall amount of data stored is very large, the unit cost of using traditional storage is very high, and the operation and maintenance is very complex when life cycle management is needed. Therefore, for the increasing amount of image data year by year, most enterprises have several pain points that are difficult to query, manage and expand.

At the same time, as the image storage service has become a part of many processes, its stability, reliability and robustness are at the same level as the core trading system. Therefore, image storage services are most concerned about reliability, consistency, scalability, throughput, and the multi-mode characteristics of unstructured storage. However, it does not focus on the ACID, HTAP and other features of the transaction.

Summary

Cloud database is an important direction of database development in the future. Cloud database architecture needs to be iterated with the requirements of cloud. In the future, the evolution of cloud database architecture will continue to develop with the change of demand.

After reading the above about the technical requirements and architecture evolution of cloud database, if you have anything else you need to know, you can find out what you are interested in in the industry information or find our professional technical engineer to answer it. Technical engineers have more than ten years of experience in the industry.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.