Shulou(Shulou.com)06/01 Report--

Data storage and processing is an ancient and important technology, from the knot notes in ancient times to the text notes of the ancients, and then to various systems after the birth of computers, until E.F.Codd put forward the relational model, human beings finally have a relatively efficient and unified data processing system-relational database.

In the traditional relational data processing system, it is customary to divide the system into online transaction processing system (OLTP) and online analytical processing (OLAP) according to business characteristics. Generally speaking, OLTP pays attention to real-time online business, requiring low delay and high throughput, and the total amount of data is generally not very large; while OLAP system is used to deal with large-scale data report analysis, requires low response time. Because of the amount of data, query requests, business requirements and the limitations of previous technical conditions, the two systems are decomposed into two independent systems, namely, the OLTP system is used to deal with online transactions, and the OLAP system is used for report processing, and the two are connected by ETL tools.

Generally speaking, these two sets of database systems are independent of each other. The common OLTP products are IBM DB2,Informix, ORACLE,SQL SERVER,mysql,PostgreSQL, etc., and the common ones in OLAP, such as TeraData,SybaseIQ,GreenPlum,HP VERTICA, SAP HANA,Hadoop big data platform and so on. In this architecture, there are two independent data systems that need to be maintained, which increases the cost of system procurement and the cost of system operation and maintenance; at the same time, the data consistency from OLTP to OLAP system in the process of ETL is also a headache.

TBase shows us a revolutionary data processing architecture, which combines OLTP and OLAP processing to complete two operations in a database system at the same time, while reducing business complexity and business costs. TBase has been running online in a provincial department for more than four years and has played an important role in the customer's architecture upgrade. at present, there are nearly 10 TBase systems running in the customer, and the cluster size is close to 100. This paper hopes to analyze the architecture of TBase with the help of customers' actual cases, so as to increase everyone's understanding of TBase.

Features of TBase architecture

TBase distributed no sharing (share nothing) distributed database. The cluster structure is as follows:

Coordinator: coordination node (CN for short), which provides external interface and is responsible for data distribution and query planning. Multiple nodes are peer-to-peer, and each node provides the same database view. Functionally, the CN only stores the global metadata of the system, not the actual business data.

Datanode: processes and stores metadata related to this node, and each node also stores fragments of business data, referred to as DN. Functionally, the DN node is responsible for completing the execution request to coordinate the distribution of the node.

GTM: global transaction manager (Global transaction manager.), which is responsible for managing cluster transaction information as well as global objects, such as sequences, of the cluster.

Under this architecture, TBase cluster has the following capabilities: multi-active / multi-master: each coordinator provides the same cluster view, which can be written from any CN, and the business does not need to be aware of the cluster topology.

Read / write expansion: data is fragmented and stored in different DN, and the read / write ability of the cluster is improved with the expansion of the cluster size.

Cluster write consistency: write transactions that occur on one CN node are presented consistently on other CN nodes as if they were on this CN node.

Cluster structure is transparent: the data is located in different database nodes. When querying data, you do not need to care that the data is located in a specific node. TBase's share nothing cluster architecture facilitates service access and reduces the threshold for service access.

HTAP capabilities of TBase

HTAP is a mixed transaction and analytical processing.

Short for Hybrid Transactional/Analytical Processing, TBase builds native HTAP capabilities through the following technologies.

Transaction ACID strongly guarantees:

In distributed database, the ACID guarantee of distributed transaction is a very challenging task, but in the process of using database, business systems often rely on the ACID capability provided by database to develop their business, so the guarantee of transaction ACID capability has also become an indispensable capability of distributed database. First take out the definition of ACID in database theory and familiarize yourself with these concepts:

ACID refers to the abbreviation of the four basic elements for the correct execution of database transactions. Includes: atomicity (Atomicity), consistency (Consistency), isolation (Isolation), persistence (Durability). A database that supports transaction (Transaction) must have these four characteristics, otherwise the correctness of the data can not be guaranteed in the transaction process (Transaction processing), and the transaction process may not meet the requirements of the trading party.

Explain in detail:

Atomicity (Atomicity): all operations in the whole transaction are either completed or not completed, and cannot be stuck in the middle. An error occurs during the execution of a transaction and is Rollback back to its state before the transaction starts, as if the transaction had never been executed.

Consistency: transactions must always keep the system in a consistent state, regardless of the number of concurrent transactions at any given time.

That is to say, if multiple transactions are concurrent, the system must also operate like serial transactions. Its main feature is protective and immutable (Preserving an Invariant). In the case of money transfer, for example, suppose there are five accounts, each account balance is 100 yuan, then the total amount of the five accounts is 500yuan. If multiple transfers occur between these five accounts at the same time, no matter how many concurrent accounts, such as 5 yuan between An and B accounts, 10 yuan between C and D accounts, and 15 yuan between B and E. The total amount of the five accounts should also be 500 yuan, which is protective and immutable.

Consistency is a major challenge in distributed transactions. Uncertainties such as network delay and operating system scheduling bring many difficulties to ensure the consistency of distributed transactions, but distributed data must have a complete set of distributed transaction consistency logic, otherwise it will bring disastrous consequences.

Isolation: transactions are executed in an isolated state so that they appear to be the only operations performed by the system at a given time. If there are two transactions that run at the same time and perform the same function, the isolation of the transaction ensures that each transaction is thought to be the only transaction in the system that is using the system. This attribute is sometimes called serialization, and to prevent confusion between transaction operations, requests must be serialized or serialized so that only one request is used for the same data at a time.

Durability: after the transaction completes, the changes made by the transaction to the database are persisted in the database and are not rolled back.

Transactions in TBase strictly abide by the transaction's ACID. Currently, two isolation levels, read committed and repeatable read, are supported, and scalable transaction throughput is provided.

There are 9 tables in the transaction test model TPC-C, which are used to simulate the delivery of orders from the warehouse, inventory status query, and a certain rollback ratio, there are 5-6 SQL in each transaction, mixed reading and writing, and strict compliance with ACID is required in the testing process. In this model, the processing capacity of TBase increases approximately linearly with the increase of cluster size, and can reach 300W tpm at the scale of 30 units (24coreJournal 64GMagne 1000Mb).

Share nothing architecture determines that with the increase of cluster size, the TPC-C processing capacity of the system will be further improved, and the processing capacity of TBase in transaction processing can reach tens of millions of QPS.

Full SQL compatibility:

SQL is a necessary tool to access the database. Today, decades after the birth of SQL, a large number of software products are developed using SQL, and every programmer uses SQL more or less. The ability of the database to deal with SQL is directly related to the stability of software products and the efficiency of program development.

According to TBase: complete SQL compatibility = perfect SQL syntax compatibility + cost optimizer + distributed executor. So TBase laid down a design rule at the beginning of the design: the SQL syntax is fully compatible with the SQL2003 standard, allowing users to use TBase like a normal database.

Therefore, in TBase, database development engineers can write SQL as they are familiar with database products before, without having to worry about the extra learning workload caused by changes in database products. Save a lot of development work for the business and reduce the cost of business upgrade.

In addition to SQL syntax compatibility, in order to achieve efficient distributed execution, TBase has a cost-based query optimizer specially designed for distributed environments, coupled with distributed executors specially designed for distributed environments.

Distributed executors provide all known database operator (operator) support, including aggregation, window functions, cube, stored procedures, custom functions, triggers, materialized views, parallel execution, and so on.

Through these designs, TBase provides a good database experience for the business, greatly reduces the business migration costs and developers' learning costs, and improves the efficiency of business migration.

SQL compatibility and performance are tested through TPC-H. The TPC-H standard meets the testing requirements in the field of data warehouse. In TPC-H, a data warehouse is implemented with 3NF, which consists of 8 basic tables and 22 queries (Q1~Q22). The main evaluation index is the response time of each query, that is, the time from the submission of the query to the return of the results (the shorter the better). The query mainly involves: multi-table association (more than 4), multi-table aggregation, sub-query and so on.

The following figure shows that the TPC-H test results of TBase in row storage mode are compared with those of international well-known database warehouses. From the test results, TBase is significantly ahead of all 22 test cases.

In database syntax, except for SQL standard, each database product has its own localization semantics. In terms of localization semantics, TBase is completely compatible with PostgreSQL syntax and partially compatible with Oracle syntax.

Full column storage capacity

There are two common physical file storage formats for databases: row storage and column storage. The following is an example of each storage structure, and the following table is our table structure and definition.

Store format by line

According to the row storage format, the data is stored in the file in the same logical order, and all the column data in a row is stored on the physical disk sequentially. The advantage of this format is obvious. If you access multiple columns of data in a row at the same time, you only need disk IO once, which is more suitable for the load of OLTP type.

Store format by column

Each column of data in the table is stored as a separate disk file. For example, the data in each column of "name", "department", "salary" and "family information" is a separate data file. This format can save a lot of disk IO than row memory when you need to access a few columns in the table at a time, which is especially efficient in aggregation scenarios, so it is often used in OLAP systems.

Row storage is the basic storage format of TBase. In order to support efficient OLAP TBase, it also provides complete column storage capacity. Business can choose the desired storage format for the data written into the database according to its own needs. TBase's column storage also supports strong compression capabilities, transparent compression and lightweight compression, transparent compression supports compression algorithms such as gzip,zstd, lightweight compression algorithms can efficiently compress according to the characteristics of data, and the compression ratio is as high as 400 +.

TPC-DS is a set of decision support system test benchmarks, mainly for the retail industry. 99 SQL queries (SQL 2003) are provided, which have a large amount of analysis data, and the test data are highly similar to the actual commercial data. At the same time, they have various business models (analysis report type, data mining type, etc.), which are mainly used to evaluate the performance of the decision support system. A working set of 1T is running on the TPC-DS stored in the TBase column, and the test results are as follows:

From the test results, the TPC-DS capability of TBase in column storage mode is much better than that of the industry benchmark.

In PS:TBase, any associated query can be made between row tables and lists, and the format can be converted to each other (through insert into select).

Resource isolation capability:

In the HTAP system, OLTP and OLAP services should run at the same time, both of them will consume a huge amount of resources. If we do not isolate the use of resources, it will inevitably cause interference between each other, the overall stability and quality of service of the video system.

Resource isolation scheme in TBase-resource isolation scheme in node group. If the OLTP and OLAP of the business access different data, you can use the node group in TBase to separate the OLTP service and OLAP service into two node groups in the cluster, set the OLTP optimizer to the CN in the OLTP node group, and strictly separate the OLAP optimization from the CN in the OLAP node group.

In TBase, each user session has a resource quota limit, which can be used to limit the CPU, memory, network and other resources of the user session. In scenarios where there is no absolute competition between OLTP and OLAP, they can also run on the same group and use user resource configuration to control and configure resources.

TBase resource isolation mode-multi-copy mode (patent technology), multi-plane technology, when the user data is still the same data, the data is copied from the OLTP system to the OLAP system by replica replication, and the conversion from row storage to column storage is realized, which separates the OLTP and OLAP services from hardware. At the same time, the reliability of data replication within the database is guaranteed by the internal mechanism of the database. The reliability is much higher than that of ETL tools. This kind of resource isolation technology in TBase is called multiplane technology.

Through the above resource isolation technologies, TBase implements the key resource isolation technologies in HTAP, which can ensure the quality of service of the system and improve customer experience.

TBase Security solution:

Thanks to TBase's perennial service to internal customers, TBase has built a set of unique data security system. It can be said that TBase is the most secure database product on the market. This feature has been unanimously recognized by internal and external customers, and has been written into the database security specification of PICC as a security technical standard.

TBase's data security system is called MLS (multiple level security). The overall view of this system is as follows:

Based on the separation of powers, the superuser rights of the system are eliminated. Add mandatory security rules to security management rules, support matrix permission division of tables, and carry out encryption and desensitization control at data access layer and storage layer to prevent data leakage during data dragging:

In retrospective and real-time audit, TBase provides complete audit rules and rich custom audit rules to support. Through FGA (fine-grained audit), TBase can also achieve real-time alarm of data unauthorized access and prevent data unauthorized access in real time.

Customer case scenario:

Case 1:TBase helps resolve the bottleneck of a provincial repository and help customers upgrade their architecture:

The original system uses a well-known database cluster. When supporting 30 billion data, the capacity of calculation and query has reached the limit, the scenario of timeout and downtime occurs frequently, and the storage rate can only reach the limit of several thousand / second, which is far from meeting the business needs.

The location of the TBase collection library in the system:

The collection library plays the role of data collection and processing in the system, not only connecting other relational databases, message middleware, etc., but also running OLTP services of some core systems, and running OLAP calculations such as model construction and offline behavior analysis on the basis of these two data. It is a typical HTAP system.

The bottleneck of the previous system is largely due to the large amount of computation of the business model itself, and the complexity of the model exceeds the processing efficiency of RAC.

After analyzing the characteristics of the business, the TBase team worked out a detailed solution for the business combined with the capabilities of TBase itself. The core point of the scheme is the multi-plane resource isolation technology of OLTP and OLAP. We divide the operation plane of the service for the request of the system in detail, and the writing of the service is all concentrated on the main plane, and the request of the query class is divided into the standby plane and the main plane according to the computational complexity and real-time requirements. Good isolation between real-time requests and offline requests. The deployment structure of the collection libraries TBase CN and DN is shown in the following figure.

At present, the collection cluster has nearly 100 nodes and stores hundreds of T of data. The OLTP business of 12 billion data can be processed in 1 second. The processing performance is far better than the previous system, but the cost is greatly reduced, and it is highly recognized by users!

Case 2: Internet of things Geographic Information system

Business background: with the advent of the Internet of things, a lot of sensor data need to be accessed. These data contain something in common and contain some point information (longitude and latitude). Combined with the correlation analysis of these location information and our existing geographic information, we can analyze the valuable data for national production and life.

The core function of this system is to correlate and query the geographic location information, which needs to be supported by the geographic information capability provided by TBase.

PS:TBase supports the most advanced open source geographic information engine PostGIS, which can provide rich and efficient geographic information processing capabilities.

The deployment architecture of the business system is as follows:

TBase upstream docks the ETL at the back end of the business sensor. After entering the cluster, the data is cleaned and transformed, and the person and location information are associated to form a wide table. GIS OLAP analysis is carried out on the basis of the wide table to output the high-value information we need.

Current GIS system scale: currently 5 machine coordination nodes + 6 machine Datanode nodes active / standby hybrid deployment.

Case 3, a core OLTP online verification system: a provincial core service Wechat official account backstage system, the core system structure is as follows:

The core TBase server of the business is deployed in the private network, and the sensitive data is deployed in this core TBase instance. An application server deployed on the Internet runs the business data related to the official account, and synchronizes the data to the business private network through the network boundary synchronization tool.

Summary

In the process of building the HTAP capability of TBase, the team members broke through one technical difficulty after another, creating one "impossible" after another. Looking back on this process, we will think of an old saying, "A long way to open up the mountains and forests"! For TBase, it should be "pioneering and innovative"!

At present, the external customers of TBase services include finance, public security, fire protection, government affairs, medical care, social security, energy, taxation and other industries, with more than 30 external clusters. The province in the case has been running online for nearly 4 years, the cluster size has gradually increased from more than a dozen at the beginning to nearly 100 now, and the number of business systems has also grown from the initial one to nearly 10 now. While helping the business solve the pain points, TBase itself has also obtained many opportunities for growth. Thank you for your attention to TBase.

Recommended reading

How does the database supporting WeChat Pay provide more than 3 million TPCC transaction processing capacity?

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.