Shulou(Shulou.com)06/01 Report--

Over the past few years, the term "microservice architecture" has been hot, describing a specific way to design software applications as independently deployable service suites. Although there is no exact definition of this architectural style, there are some common characteristics around business capabilities, automated deployment, network intelligence, and decentralized control of language and data.

In short, a microservice architecture is a way to develop a single application as a set of small services, each running in its own process and communicating with lightweight mechanisms (usually the API of HTTP resources). These services are built around business functions and can be deployed independently through a fully automated deployment mechanism. These microservices minimize centralized management, while microservices can be written in different programming languages and use different data storage technologies.

When it comes to data storage technology, we have to talk about databases, but in fact, there is a subtle relationship between micro-services and databases, and micro-services have different requirements for databases than traditional architectures. what is the relationship between micro-services and databases? What is the impact of databases on microservices? How to choose a database suitable for micro-service? Wang Tao, co-founder and CTO of Giant Sequoia Database, shared his views with CSDN reporters.

Micro-service architecture gives birth to distributed database

Wang Tao believes that talking about databases must be inseparable from applications. From the perspective of application development, the application development within many enterprises is transforming from the traditional "chimney" development of middleware and database to micro-service architecture. In the micro-service architecture, almost every micro-service needs the ability to provide data persistence, and users also hope that the amount of data carried by each micro-service can be expanded infinitely. However, in the process of using micro-service architecture, each micro-service uses its own independent database storage, which will make the data concentrated in one place in the past scattered to many different devices, resulting in serious fragmentation of the data of the whole IT architecture. For example, some Internet companies maintain 20,000 to 30,000 MySQL databases in the production system alone, so it is extremely difficult to integrate data within the enterprise.

In fact, when enterprise users adopted the micro-service architecture, there were two approaches in the industry from the perspective of data management.

The first approach is to micro-service the application, and the underlying database is stored in a traditional centralized database. This method is relatively small for the transformation of the application, and the learning cost is also low for the DBA operation and maintenance personnel, but correspondingly, it has some problems, such as tight data coupling, inelastic expansion, and possible single point of failure.

The second approach, which may also be widely used in the industry, is that each group of micro-services corresponds to a separate small database, often using MySQL or PostgreSQL. This mechanism can solve the problem of centralized storage, but it also brings new challenges, including extremely fragmented data, unable to share among micro-services, and extremely high cost of operation and maintenance.

Therefore, neither of the two methods can solve the problem of data storage management under micro-service, so the distributed database is to solve the above two problems. The first is to achieve flexible expansion of data for each micro-service, and the second is to achieve unified data management for the entire enterprise IT to avoid fragmented storage.

Create a distributed database suitable for micro-service

In order to build a database suitable for micro-service architecture, Giant Sequoia database adopts the architecture of computing and storage separation. Among them, the storage layer uses a self-developed native distributed database engine, while the upper computing layer can create hundreds of database instances, and each database instance is completely transparent to the application and does not need to be perceived.

Therefore, under this system architecture, from the point of view of a single application, it is completely consistent with the traditional standard database, and there is no need to pay attention to which different physical devices the data is divided into. At the same time, all physical devices are managed logically, and even the data in different instances can be shared under configurable permissions.

So, what characteristics should a distributed database suitable for micro-services have? Wang Tao believes that this should be seen mainly from two dimensions and five aspects.

The first is the compatibility of traditional technology, and the other is the innovation of technology and architecture.

In terms of compatibility with traditional technologies, first of all, ACID must be supported. Because from a database point of view, although many people say that CAP can't have both at the expense of consistency, Giant Sequoia believes that this is not desirable. For most companies, data is the core lifeline, we must not sacrifice the consistency and security of data for the sake of distribution, and we need to be responsible for the property and information of users. Therefore, the new distributed database for online transactions must have perfect support for traditional ACID and be compatible with the data security consistency of traditional Oracle DB2.

Secondly, the integrity of SQL. This is mainly from the perspective of compatibility with traditional applications and reuse of developers' capabilities. Generally speaking, the integrity of SQL syntax compatibility and compatibility with existing standards, such as MySQL, Oracle, DB2, and PostgreSQL, are required.

From the forward-looking point of view of the new technology, first of all, the future is the era of private cloud and micro-service applications, so as a distributed database, it is not simply positioned as a substitute for a database in the past. The core value of distributed database is that it can provide database access to hundreds of small services separated from chimney architecture to micro-service architecture in the form of service resource pool at the database level. Under this positioning, the database resource pool must meet the distributed elastic expansion on the basis of ensuring 100% compatibility with the traditional database. When there is insufficient space and computing capacity in the resource pool, the capacity needs to be expanded by dynamically adding computing storage nodes.

Secondly, in the past, the database only aimed at a specific application and adopted the way of one-to-one binding between middleware and database, so it only needed to provide its own access mode. However, when pooling database resources, the upper applications naturally face services from different developers, different business types, and different SLA levels, and they adopt different development processes, SQL standards, and security policies, so distributed databases must be able to support multiple mode access interfaces.

Finally, HTAP, that is, transaction analysis hybrid processing capacity. For example, for some accounting data, the core key application may come from the real-time use of these data by online trading business, but at the same time, some back-end real-time reports, or when security auditors need to conduct statistical analysis, businesses from different microservices may need to access the same data in the form of transaction and analysis at the same time. In this case, it is particularly important to isolate the physical resources of the transaction and analysis business in the resource pool, access the same data at the same time and do not interfere with each other at the same time, therefore, databases suitable for micro-services must have strong transaction analysis mixed processing capabilities.

Giant sequoia database, distributed database suitable for micro-service

Just like the technical orientation and goal of Giant Sequoia for distributed database, Giant Sequoia database SequoiaDB itself is built on two layers: distributed storage base and upper database instance. As a resource pool, the underlying distributed storage is responsible for data storage, distributed transaction control, recording and table locking, all of which are implemented in the underlying native distributed storage.

The database instance layer provides SQL services for upper-layer applications. Users can create structured instances such as MySQL, PostgreSQL, Spark SQL, or unstructured instances of JSON or S3 file systems. The data in each instance is completely transparent to the upper application. Therefore, in SequoiaDB, a MySQL table can easily store billions or even tens of billions of data, and developers do not need to pay attention to how many physical devices the underlying tables are scattered when writing SQL.

As the representative of native distributed database and new generation distributed database in the industry, SequoiaDB is fully compatible with traditional technology for distributed transaction and ACID, and its architecture and functional features are fully compatible with traditional database. At the same time, SequoiaDB also actively embraces the new generation of micro-service and cloud computing framework, supporting a series of features such as elastic expansion, resource isolation, multi-tenancy, configurable consistency, multi-mode (supporting all kinds of SQL protocols), and configurable disaster recovery strategy within the cluster when facing micro-service application development and cloud computing infrastructure.

In fact, the capacity bottleneck of traditional single point database is only one of the problems solved by distributed database. More importantly, under the trend of micro-service application development and cloud platform in the future, applications are no longer built with "chimney-like" middleware plus database mode, but a complex mesh model built by thousands or even tens of thousands of micro-service programs. Therefore, distributed database needs to be able to meet the needs of flexible expansion, high concurrency, high throughput, and agility of upper-level applications. SequoiaDB has an excellent performance in these areas, including:

Complete ACID support, transaction and consistency guarantee; complete support of SQL, traditional database MySQL/PostgreSQL syntax is fully compatible. Distributed and expansible, in response to changes in the amount of data, to achieve flexible expansion of the storage layer and computing layer; multi-mode access interface to support multi-type data management and multi-mode access interface; HTAP transaction / analysis mixed processing capability to achieve physical isolation of data without interference with each other under complex business requirements.

In the latest version 3.2 released by the conference, Giant Sequoia has greatly optimized and improved the performance of SequoiaDB, so that its overall performance has been improved by 2 to 3 times and its CPU consumption has been saved by more than 30% under the distributed transactional business, thus greatly increasing its support for micro-services.

SequoiaDB doesn't just support microservices.

In fact, SequoiaDB is not just a mentor of micro-services, its larger dimension is a true financial-level distributed relational database.

At present, the application scenarios of giant sequoia database in enterprise mainly include distributed online transactions, data center and distributed content management.

Online transaction is one of the most widely used scenarios of database, which is usually used to support core business operations. The core business value of distributed online transaction database includes distributed architecture transformation, high concurrency, high processing capacity, continuous business expansion and autonomous controllability and data security requirements. SequoiaDB storage engine adopts native distributed architecture with easy expansion, complete support for distributed transactions, strong consistency and high availability of multiple copies, and no single point of failure, and database engine natively supports multi-center disaster recovery.

Data center platform is a very hot concept at present. Data center platform plays a very important role in enterprise micro-service architecture. It connects the high-level fast iterative micro-service applications and lower-level infrastructure like a gear. At the same time, it can also provide real-time online services for all data, generally referring to all external services except traditional core transactions. The core business values of data center platform based on SequoiaDB include: high-performance real-time access to data. The massive data is online in the whole life cycle, and the business continues to expand.

Content management platform provides enterprises with the ability to store, manage and use large amounts of unstructured data. Common applications include image platform, document management platform, audio and video recording system and so on. The core business value of the content management platform based on SequoiaDB includes massive unstructured data management and real-time access, rich content management functions, massive unstructured data lifecycle online and business continuous expansion.

It is reported that at present, the giant sequoia database has been launched in the core production business of nearly 100 large commercial banks, and is widely used in finance, telecommunications, government, Internet, transportation and other fields, with a total of more than 1000 enterprise users. At the same time, Giant Sequoia is also the first database manufacturer in China to be selected as a Gartner database report for two consecutive years.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.