Shulou(Shulou.com)06/01 Report--

TiDB overall architecture and what is the role, I believe that many inexperienced people are helpless about this, this article summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

As far as I know, many companies are currently using TiDB in production environments, such as Xiaomi, Xiaohongshu, Hungry, Meituan, etc.

Nowadays, the cost performance of hardware is getting higher and higher, the network transmission speed is getting faster and faster, and the trend of database layering is gradually emerging. People no longer insist on using one solution to solve all storage problems, but through layering, cache and database are responsible for their own business scenarios.

The current database field faces various problems, such as scaling, consistency, big data analysis, integration with cloud infrastructure, etc. There are many problems. The existing database solutions and big data analysis engine solutions are basically in a state of fragmentation. Since Oracle and MySQL databases are not designed for distributed environments, even if they are fragmented at the database level by means of sub-database, sub-table or middleware, Essentially, it just replicates the same stack, rather than optimizing storage and compute for distributed systems, which is the essence of why cross-business queries or cross-physical queries and writes are cumbersome. Although NoSQL solves the problem of database elastic expansion, it gives up strong consistency of data and support for ACID transactions, which brings new problems.

In order to solve this problem, TiDB abstracts and separates the computing and storage layers to a high degree in architecture, and makes it possible for mixed load scenarios through IO priority queues, intelligent replica scheduling, and mixed row and column storage. TiDB, as an open source distributed relational database, is characterized by almost 100% compatibility with MySQL interfaces, MySQL syntax and protocols, Auto Scaling without losing ACID transactions, high availability, simultaneous OLTP and OLAP workloads, and no need for ETL.

TiDB overall architecture diagram

The overall architecture of the TiDB product is highly layered, consisting of a distributed SQL layer (TiDB), a distributed KV storage engine (TiKV), and PD modules that manage the entire cluster. Infinite horizontal expansion is a major feature of TiDB, which includes two aspects: computing power and storage capacity.

HTAP provides developers with a new idea of real-time data analysis, without the need to maintain another offline data warehouse, which not only reduces the ETL work, but also saves a large part of the storage and computing costs used to build a data warehouse. HTAP will be an important trend in the future. Huang Dongxu introduced the four main application scenarios of TiDB, one is MySQL fragmentation and merging; the other is to directly replace MySQL; the third is to be used as a data warehouse; and the fourth is to be used as a module of other systems.

Case 1: MySQL fragmentation and merging

Syncer

The first type of scenario for TiDB applications is MySQL fragmentation and merging. For businesses that already use MySQL, sub-database, sub-table, fragmentation, and middleware are common methods. With the increase of fragmentation, cross-fragmentation queries are a big problem. TiDB is compatible with MySQL access protocol at the business layer. PingCAP makes a data synchronization tool-Syncer, which can take TiDB as a MySQL Slave and connect TiDB as a slave database of existing database behind the master MySQL database. At this layer, data can be connected, and complex cross-database, cross-table, and cross-business real-time SQL queries can be directly performed. Huang Dongxu mentioned,"In the past, databases were all multi-master and multi-slave. With TiDB, multi-master and multi-slave can be achieved in reverse. "

Case 2: Replace MySQL directly

The second scenario is to replace MySQL directly with TiDB. If your IT architecture did not consider the problem of sub-database and sub-table at the beginning of the construction, all of them used MySQL, with the rapid growth of the business, there are more and more massive OLTP scenarios with high concurrency, how to solve the shortcomings of the architecture?

On a TiDB database, all business scenarios do not need to be divided into databases and tables, and all distributed work is done by the database layer. TiDB is compatible with MySQL protocol, so it can directly replace MySQL, and basically out of the box, do not have to worry about the heavy workload and complex maintenance cost brought by the traditional database and table separation scheme, friendly user interface allows regular technicians to efficiently maintain and manage. In addition, TiDB has similar expansion capability to NoSQL, which can improve the service support capability of the system through horizontal expansion under the condition that the data volume and access traffic continue to grow, and the response delay is stable.

Huang Dongxu mentioned the case of motorcycle in his speech. All the databases in the early days of motorcycle used MySQL. With the rapid growth of business, the disadvantages of MySQL gradually appeared. Motorcycle began to replace MySQL with TiDB in early 2017. Today, Mobai's IT system has deployed several TiDB clusters, nearly 100 nodes, carrying dozens of terabytes of various types of data.

Case 3: Data Warehouse

TiDB itself is a distributed system, and the third usage scenario is to use TiDB as a data warehouse. TPC-H is a test set in the field of data analysis. TiDB 2.0 has greatly improved its performance in OLAP scenarios. Some complex queries that can only be run in the data warehouse can basically be controlled within 10 seconds in TiDB 2.0. Of course, because the scope of OLAP is very large, TiDB SQL is also uncertain, for this reason PingCAP open source TiSpark, TiSpark is a Spark plug-in, users can directly use Spark SQL in real time on TiKV big data analysis.

Case 4: As a module of another system

TiDB is a traditional storage and computation separation project, its underlying Key-Value layer, can be used as a Replacement of HBase alone, it also supports cross-line transactions. TiDB provides two API interfaces, one is ACID Transaction API, which is used to support cross-line transactions; the other is Raw API, which can do single-line transactions, in exchange for the overall performance improvement, but does not provide ACID support for cross-line transactions. Users can choose between the two APIs according to their own needs. For example, some users have implemented the Redis protocol directly on top of TiKV, replacing TiKV with some Redis scenarios with large capacity and low latency requirements.

After reading the above content, do you understand the overall architecture of TiDB and what its role is? If you still want to learn more skills or want to know more related content, welcome to pay attention to the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.