Shulou(Shulou.com)06/01 Report--

This article mainly introduces "what are the new features of TiDB DM2.0GA". In daily operation, I believe many people have doubts about the new features of TiDB DM2.0GA. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful for you to answer the doubts about "what are the new features of TiDB DM2.0GA?" Next, please follow the editor to study!

In the development process of social digitization and intelligence, massive data brings great challenges, various industries are accelerating digital transformation, more and more enterprises realize that data infrastructure is the key to success. However, as the core of data infrastructure, traditional databases such as MySQL face performance and capacity bottlenecks. The scheme of database partitioning and table partitioning implemented by middleware is complex and brings high operation and maintenance costs.

As an enterprise-level NewSQL database, TiDB adopts the architecture of separation of computing and storage, and can be flexibly expanded according to business needs to meet the needs of more real-time and intelligent data applications. TiDB provides Data Migration (DM) ecological tools to help users migrate data from MySQL to TiDB, effectively reducing migration costs and risks.

DM is an integrated data migration task management platform developed by PingCAP, which supports full data migration and incremental data replication from MySQL, Aurora or MariaDB to TiDB. DM 2.0 has been officially released, adding enterprise-level features such as high availability, merging and migration of sub-databases and tables in optimistic coordination mode, and bringing a series of improvements in ease of use, ensuring that users' original database can be smoothly switched to TiDB, without having to worry about failures and data loss caused by migration.

# # New Features of DM 2.0

High availability of data migration tasks

DM 2.0 provides high availability of data migration tasks, and some DM-master and DM-worker nodes can still ensure the normal operation of data migration tasks even if they are abnormal.

When multiple DM-master nodes are deployed, all DM-master nodes are clustered using embedded etcd. The DM-master cluster is used to store cluster node information, task configuration and other metadata, and the leader node is selected through etcd. The leader node is used to provide various services related to cluster management and data migration task management. If the number of available DM-master nodes is more than half of the deployment nodes, the service can be provided normally.

When the number of deployed DM-worker nodes exceeds the number of upstream MySQL/MariaDB nodes, the relevant DM-worker nodes that exceed the number of upstream nodes will be idle by default. If a DM-worker node goes offline or is isolated from DM-master, DM-master can automatically schedule the data migration tasks related to the original DM-worker node to other idle DM-worker nodes and continue to run.

# merge and migration of sub-libraries and tables in optimistic coordination mode

DM version 1.0 supports online execution of DDL statements (commonly known as Sharding DDL) for database and table partitions. By using the pessimistic mode, that is, when an upstream sub-table executes a certain DDL, the migration of that sub-table will be paused until all other sub-tables have executed the same DDL before executing the DDL downstream and continuing the data migration. The advantage of the pessimistic coordination mode is that it can ensure that the data migrated downstream will not go wrong, while the disadvantage is that it will pause the data migration, which is not conducive to the grayscale change to the upstream, and significantly increase the delay of incremental data replication.

DM version 2.0 provides a new optimistic coordination mode. The DDL executed on one sub-table is automatically modified to be compatible with other sub-tables and immediately applied downstream, which will not prevent the migration of DML executed by any sub-table. The optimistic coordination mode is suitable for upstream gray update, release scenarios, or scenarios that are sensitive to synchronization delays in the process of upstream database table structure changes.

In optimistic coordination mode, when DM-worker receives the DDL from upstream, it forwards the updated table structure to DM-master. DM-worker tracks the current table structure of each sub-table, DM-master merges to be compatible with the composite structure from each sub-table DML, and then notifies the corresponding DM-worker to migrate the corresponding DDL downstream; for DML, it migrates directly downstream.

DM version 2.0 tentatively supports the migration of data from MySQL 8.0 to TiDB, while providing TLS support to build a three-dimensional data security system to ensure the security and compliance of connections and transmission between DM components and between DM components and upstream and downstream databases, and to help enterprises achieve no loss, disclosure, tampering and privacy compliance in the whole life cycle.

# overall improvement of ease of use

In addition to the new features, DM 2.0 brings an overall improvement in ease of use. Users can deploy and operate DM 2.0 through TiUP. At the same time, they can use TiUP to import and upgrade version 1.0 DM to version 2.0. In DM 2.0, DM-worker uses API provided by DM-master to register dynamically. When expanding and reducing DM-worker, it is no longer necessary to restart DM-master components, which effectively improves business continuity.

For hosted MySQL provided by cloud vendors such as AWS Aurora and Aliyun RDS, users are usually unable to obtain SUPER permission and thus cannot obtain consistent snapshots when full data is exported. In DM 2.0, by recording the binlog position range from the beginning to the end of the full export process and automatically ensuring the opening of safe-mode in the incremental phase, the final consistency of data is ensured without manual processing by the user. DM 2.0 also provides better compatibility support during permission checking for specific permissions in Aurora, such as "SELECT INTO S3".

In DM 2.0, the query-status command can not only query the possible data migration exceptions, but also provide "Workaround" information to guide users on how to deal with some common exceptions. DM 2.0 introduces handle-error commands to replace sql-skip and sql-replace commands in DM 1.0, which simplifies the process of dealing with faulty SQL statements during data migration.

In addition, DM 2.0 adds automatic processing of the corresponding sql_mode in fully exported data and incremental binlog data, ensuring that manual configuration and intervention are minimized. DM 2.0 also enhances the ease of use of a series of features, including automatic cleaning of all exported files, configuration parameter optimization, monitoring panel optimization, log display optimization, and so on.

# # user practice

# WeBank

WeBank was granted a financial license issued by Shenzhen Banking Regulatory Bureau in December 2014. it is the first private bank in China initiated by Tencent and other well-known enterprises. It is committed to providing differentiated, distinctive, high-quality and convenient financial services for the general public and small enterprises.

WeBank uses TiDB in multiple business scenarios, in which batch tasks and pipeline log archiving are highly dependent on the sub-table and combined table feature of DM. In batch task scenarios, DM is used to summarize the isomorphic sub-tables of upstream MySQL instances to downstream TiDB, and then use the horizontal expansion capability of TiDB to improve batch efficiency. In the pipeline log archiving scenario, DM is also used to summarize the isomorphic sub-tables of upstream MySQL instances into TiDB, and the horizontal scalability of TiDB is used to provide theoretical unlimited storage capacity.

The original version of DM 1.0 encountered some problems in the process of use: after the abnormal hang of the Worker component of DM, it will lead to the suspension of data synchronization, which requires manual intervention to recover, and the operation is tedious and will affect the timeliness of data synchronization. Secondly, in the financial scenario, the grayscale strategy is generally used to change the table structure, that is, for the isomorphic sub-tables of multiple upstream MySQL instances, one of the instances will be changed at gray level, and only after observing that there are no anomalies for a few days will continue to change the table structure of the remaining isomorphic sub-tables. This scenario will lead to data synchronization block live under DM 1.0, which will also affect the timeliness of data synchronization.

In view of the lack of some functions of DM 1.0 in the actual scene, WeBank database team excavated and refined the requirements through business POC testing, carried out in-depth scheme discussion with PingCAP, and carried out a series of functional development and optimization work. The version of DM 2.0 already covers enterprise-level features such as high availability of components and support for grayscale changes, and can meet the needs of financial-level data synchronization. In addition, DM 2.0 also has a large number of optimizations in terms of ease of use, such as using TiUP to deploy and maintain multiple DM clusters more easily, simplifying the upstream source configuration information of Worker, and making error messages clearer and easier to read.

# ideal car

Ideal Automobile is committed to developing smart electric vehicles that are better than fuel cars. Since the first ideal ONE was officially launched in November 2019, ideal cars have delivered 20000 vehicles in only 10 months, setting a record for the fastest delivery of new car-making forces in China. Micro-service has become the basis of digital transformation and upgrading of enterprises in the cloud native era. At present, more than 99% of ideal cars are accumulated, and more than 400 + business applications are built on micro-services, covering car networking, order mall, vehicle production, after-sales, logistics and other business processes. In the micro-service architecture, each individual micro-service corresponds to an independent MySQL database (based on public cloud RDS). Ideal Automobile uses TiDB Data Migration (DM) tool to synchronize the data of multiple MySQL databases to a set of TiDB clusters in real time to meet the application requirements of the two business scenarios.

On the one hand, TiDB meets the needs of real-time data checking across multiple MySQL databases, making use of the HTAP capability of TiFlash to provide real-time business analysis reports. On the other hand, TiDB is used to back up multiple MySQL databases in public cloud in real time, which not only improves business availability, but also reduces the cost of slave resources needed to achieve load balancing in public cloud RDS under read-write separation scenarios.

Based on the strong dependence of the business on DM tools, ideal Motors upgraded the original DM 1.0 cluster to DM 2.0 through TiUP, and deeply tested the high availability features of DM 2.0, including the high availability of DM-master and DM-worker nodes, automatic scheduling and correctness assurance of data migration tasks, and DM-master expansion after upgrading from 1.0 to 2.0. In general, DM 2.0 reduces the risk of real-time data synchronization from MySQL to TiDB, ensuring no data loss and high service availability during synchronization.

At this point, the study on "what are the new features of TiDB DM2.0GA" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.