Shulou(Shulou.com)05/31 Report--

This article introduces the relevant knowledge of "what is the importance of distributed database". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

I remember that two or three years ago, when people were talking about technologies such as distributed database, they often used words such as "future" to describe the application prospect of this new technology. In 2020, the new generation of database leader Snowflake was successfully listed and became the largest software company in history, IPO. When we looked back, we found that the future had come!

However, how to correctly land the distributed database in the enterprise has always been the focus of discussion in the industry. In China, when most readers learn about distributed database for the first time, the first question is: can distributed database replace Oracle? However, from the direction of the development of global data volume, its explosive growth is mainly focused on diversified business scenarios based on digital innovation. Therefore, simply replacing the traditional Oracle to occupy the inherent field of core advantage is not the growth direction of the database in the future. Take Snowflake as an example, its business does not replace the core application field of Oracle, but returns $70 billion with revenue of $400m, which is shaking the leading position of Oracle in the data market. The reason behind this is worth thinking about.

Thinking based on "substitution" can never be "surpassed".

In fact, there is no "yes" or "no" across-the-board answer to the question of replacing Oracle. The original purpose of distributed database design is to solve new practical business problems. In the scenario that Oracle can not meet, we should meet the opportunity of digital transformation with enterprise customers, not simply to replace an existing system.

Traditional relational database has been deeply cultivated in core transactions and other fields for more than 40 years. So far, most pure transaction scenarios have no essential change in terms of data volume or business model, and their business expansion space is also very limited. In the process of enterprise digital transformation, the amount of data will expand rapidly with the development of business, forming a new business demand and data increment, which will bring new market opportunities for the database.

Compared with traditional relational database, distributed database not only provides the ability of ACID transaction consistency, but also has more flexible expansion ability and multi-data model processing ability. In order to meet the emerging business requirements of elastic expansion of massive data, it is our best practice to apply distributed architecture in the industry to "choose distributed database instead of Oracle". In other words, the use of distributed database step by step iteration, with the new digital business gradually infiltrating into the traditional business, become a new data core scene, is the best landing scheme of distributed database technology in the enterprise.

Therefore, the star sea of distributed database is not just a simple replacement of traditional relational database. If the replacement of the inherent architecture is just for the purpose of using and popularizing new technologies, it will face great technical risks and challenges. Only from the point of view of enterprise customers and working with customers to explore the new data value in the digital transformation can we break through the inherent framework and establish a new distributed technology track beyond the traditional architectural boundaries.

How to choose the best landing scene

From the perspective of business scenarios, Oracle, DB2 and other databases have experienced more than 40 years of development history since the birth of relational databases in the late 1970s. For their inherent business scenarios, they have basically achieved the best in the industry. However, the traditional transactional database is obviously inadequate for the new data, micro-service data fusion management, real-time access to massive data, unstructured online processing and so on. When enterprise customers choose distributed database landing scenarios, they should choose appropriate application scenarios to give full play to their advantages and capabilities, and continue to polish the operation and maintenance capabilities of the technical team, and gradually push them to the core.

1) data of China-Taiwan Union Machine and Lake Warehouse

In many enterprise IT architecture planning, the data center has become a part of the whole IT strategy, including historical data platform and even unstructured data processing and other multi-data model areas, covering almost all data processing and service capabilities in the enterprise except business application systems.

In this scenario, Oracle cannot meet the scalability, while Hadoop cannot support real-time concurrent services. In the world, there is no technical system directly opposite to it, and the closest one is Lakehouse (lake-warehouse integration). The main manufacturers of Lakehouse industry include distributed database manufacturers such as Snowflake and Databricks, whose products can be divided into two core modules: data lake and computing engine. In 2020, Gartner further introduced the Augmented Transactions Processing processing scenario, which emphasizes transaction consistency and requires the database to maintain low latency in the process of analysis and processing, in order to improve the ability of real-time online processing.

It can be predicted that the online lake warehouse, which supports Augmented Transactions Processing capability, will enhance the real-time online processing capability for the data center, realize the simultaneous use of multi-business and multi-data models for data storage, accelerate data processing efficiency, reduce data redundancy, and provide a greener data infrastructure.

2) data fusion management of microservices

Today, when the micro-service application development architecture has gradually become the mainstream, the traditional architecture of one application to one database has been broken down into dozens of hundreds of micro-services, each of which may need to use independent database instances. therefore, the number of database instances in the enterprise has shown a blowout trend in recent years.

The emergence of distributed database can solve the problems of expansion and maintenance in the batch management of database instances. At the same time, based on engine-level multi-mode technology, distributed database can support online transactions of multiple database engines based on the same data, and achieve ACID consistency of heterogeneous data sources under micro-service architecture through cross-engine transaction consistency. Therefore, compared with traditional Oracle and other databases, distributed database technology is more conducive to micro-service, open up the underlying data for enterprises, and reduce the cost of data storage and management. Assist the R & D team to deliver DevOps continuously and improve the efficiency of product research and development.

3) Real-time access to massive data

The storage and calculation of massive data is usually completed by the data warehouse (MPP database) or big data platform (Hadoop), and the amount of data often reaches the level of hundreds of billions (or even trillions). In the traditional application, because the data warehouse needs to be cleaned and stored in advance, the data warehouse and big data platform can not support real-time concurrent data access, which is limited to the processing model of the existing platform, so it is difficult to innovate the online business. However, in the process of data transformation, solutions that require online real-time processing of massive data will be produced in the customer online transaction, historical data service platform or IoT Internet of things system.

Distributed databases can help enterprise customers get a better experience in this scenario. First of all, distributed database has the same flexible scalability as Hadoop and data warehouse. Secondly, distributed database can provide the same ACID support as traditional relational database to ensure the transaction consistency of its key business. Most importantly, distributed databases can better support highly concurrent business access, and can achieve millisecond data retrieval in tables containing hundreds of billions or even trillions of records, just like using a stand-alone database.

4) unstructured data governance

Unstructured data, including pictures, documents, audio and video and other object files, used to be simply stored in the storage system, providing a single storage and access function. Therefore, in addition to the business systems that directly operate these files, unstructured data is a black box for other systems in the enterprise, unable to give full play to the potential value of the data.

Today's business systems often begin to use this kind of unstructured data online on a large scale. For example, in the collection of all kinds of documents in the business, the retention of the original files required by the supervision of avatars, fingerprints and voiceprints in the transaction process, and the 360 customer portrait systems of all kinds of business, the processing process requires high-frequency comparison with unstructured data, concurrent processing and sampling correction, these scenarios all require online real-time management of unstructured data. Simply using NAS or network disk to store large amounts of unstructured data has long been unable to meet the needs of this kind of real-time online processing capabilities.

At the same time, in the digital transformation, unstructured data is no longer a static file. Through AI machine learning and comparison analysis, unstructured data will contain more diversified business attributes and provide information input for all kinds of business systems. Therefore, it is necessary to carry out effective classified governance to activate the potential value of unstructured data assets.

Distributed database can effectively improve the real-time processing ability of unstructured data, combined with engine-level multi-mode ability to store structured and object data, and can effectively realize the classification and governance of label-based feature data. become an enterprise "unstructured data governance" to establish a solid base.

Evolution trend of distributed Technology

From a technical point of view, in the demand for the rapid development of massive data and Internet applications in various industries, elastic expansion, multi-mode and other functions are difficult to meet by the traditional Oracle database, and it is also the greatest value and purpose of distributed database. In the context of this kind of technology, "choose distributed database instead of Oracle" is the most correct answer. The best landing and use of distributed database is the step-by-step iterative process from massive data business to core. Starting with the emerging business requirements of massive data elastic expansion, with the deepening of business innovation, gradually infiltrated into the traditional business and applications.

1) flexibility: separation of deposits and accounts to achieve flexible expansion

As a distributed database, flexible scalability is the core meaning and value of its existence. Compared with the traditional MPP data warehouse, the new distributed database can realize the independent expansion of storage and computing resources based on the deployment model of separation of storage and calculation, and achieve flexible expansion on demand without awareness of the application level.

2) transaction: native distributed strong consistency

As the distributed technology gradually approaches the core of the business, customers' requirements for ACID transaction consistency continue to improve. For example, in online trading business, the ability of "RR-level transaction isolation" is often required. In such requirements, solutions based on sub-library and sub-table technology, because the database itself can not provide this support, (some products do not even provide transaction support or weakening through 1PC commit). As a result, a large number of peripheral application logic cooperation is needed to achieve the final consistency effect, which consumes a lot of developers' design energy. As for the native distributed database, thanks to the distributed design derived from the kernel, customers can safely hand over the transaction consistency logic to the database layer for processing, allowing developers to return to pure business design, provide direct and effective R & D output for the business, and improve the R & D efficiency of enterprises.

3) Fusion: engine-level multi-mode, opening a new track in Hu Cang.

After more than 40 years of development, relational database has derived the ability to support XML, JSON, geographic information, map and other different capabilities from the original pure structured model. Due to the use of isomorphic engine on the same physical device, the multi-mode capability of traditional database is difficult to be brought into full play. In the distributed database architecture, users can use different physical devices and underlying data structures to carry the computing and storage engines of different data models, and really achieve the native engine-level multi-mode technology. Thus, it provides data sharing across different data models and even different database languages and engines, and avoids transmission delay and waste of storage space due to frequent data copying when online processing between different models. Based on the multi-mode ability to build a data lake that simultaneously meets structured, semi-structured and unstructured data, while giving cross-engine data consistency and real-time data analysis capabilities, in a real sense, the global data is visible in real time. As a result, developers can bridge the development gap between different data engines, improve development efficiency and system performance, and open up a new track of distributed technology.

Summary

After more than 40 years of development, the traditional relational database has almost reached the extreme in its main core transaction field. Using the core transaction scenario alone to mark the new distributed database is like using the traditional carriage standard to measure the new automobile technology, which can not make a reasonable evaluation of the new technology.

First of all, the birth of distributed database is to solve the scene that traditional database is not good at, and it also takes a long time to perfect in the field of relational database. Thanks to the characteristics of high flexibility, strong transaction consistency and multi-mode fusion, in recent years, many enterprises have been in the fields of connecting data to Taiwan, micro-service data fusion management, real-time access to massive data, unstructured online processing and other fields. to achieve large-scale production of native distributed databases. We are pleased to see that the application field of distributed database has been greatly expanded almost every year, and it has become an indispensable flexible data infrastructure to support the digital reform and upgrading of enterprises.

Giant Sequoia Database has released support based on multi-mode engines since 2014, providing customers with a distributed data infrastructure that can manage multiple data structures at the same time. It has assisted more than 100 financial bank customers and more than 1000 enterprise users to provide distributed database technology. With the self-patented STP distributed sequence clock protocol, RR-level transaction isolation and cross-engine transaction consistency are realized, and the best practice of online lake warehouse production landing is provided for the data center. It has successfully helped customers to provide a secure, stable, flexible, scalable, high-performance and high-concurrency data base in a production environment with up to 1.2 trillion data volumes.

Looking back on the past 10 years, distributed database has experienced from industry questioning, small-scale testing, to large-scale application in some industries. We firmly believe that in the new year, China's distributed database industry will develop more vigorously, and the application scale of distributed database in the next three to five years is expected to surpass Oracle to become an important part of the core trading business.

In the future, with the help of 100% self-developed native distributed database engine and engine-level multi-mode features, Giant Sequoia Database will uphold customer-centric values and work with customers and upstream and downstream partners to provide quality products, technical services and ecological support for financial, energy, operators, government and enterprise customers to promote the global digitization process.

Distributed database: the future has come.

This is the end of the content of "what is the importance of distributed databases". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.