Shulou(Shulou.com)05/31 Report--

Open source distributed database RadonDB core technology and implementation of what is, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain in detail for you, people with this need can come to learn, hope you can gain something.

RadonDB is a new generation of distributed relational database that combines distributed consistency protocol Raft with MySQL. It has the advantages of both NewSQL and MySQL databases. The following will be from the architecture, implementation, high availability and other points of view, combined with open source code for everyone in-depth analysis of the core technology and implementation of RadonDB.

RadonDB is an open source distributed database. Why is it called RadonDB? Radon in RadonDB comes from the periodic table and is an inert gas called radon. Because of its stable chemical properties, we named this database product after it.

RadonDB consists of two parts, Radon and Xenon, and is not a simple database middleware. Among them, Radon is similar to a database middleware, and Xenon is a highly available MySQL management cluster tool.

The figure above shows the entire architecture of RadonDB, in which the top layer is Radon, a distributed SQL layer responsible for SQL parsing and forwarding. This layer is what we call database middleware, which generates a distributed execution plan for SQL statements according to user requests and pushes them to the following storage layer.

The following layer is Xenon, a highly available MySQL management tool. In each dotted frame of this layer, there is a "master" and "two slaves" MySQL, which is managed through Xenon. Xenon is actually a non-centralized management tool. When the master node dies, it will use Raft protocol to select the master, select the new "master", and then synchronize the data according to these mechanisms of MySQL Binlog, so that the new master node will continue to serve.

Let's talk about some technical details of RadonDB. The main function of RadonDB is to generate a distributed plan for the user SQL and execute the executor in parallel. When the actuator is sent to the storage layer, it carries out secondary operations such as ORDER BY, LIMIT, GROUP BY and so on.

Radon supports cluster mode, so it is basically stateless. When one of the nodes dies, the other nodes can be quickly migrated to ensure the high availability of the Radon layer.

If you look at the specific workflow of Radon from the code level, after the user SQL arrives at Radon, the query.go file will generate the syntax tree for SQL, and then process it according to the type of SQL.

First, a distributed execution plan is generated according to the syntax tree. The distributed execution plan is to find out which backend each specific data is distributed in according to the routing table, and then generate specific clauses.

After the distributed execution plan is generated, the Insert executor file is run and sent to the corresponding node for execution.

These are the basic working mechanisms of the Radon layer in RadonDB.

The Radon layer also has an important function-distributed transactions. Radon distributed transactions are based on MySQL native transactions-XA transactions. The XA transaction of MySQL can be divided into five stages: the first stage is XA START, the second stage is SQL execution, the third stage is XA END, and the fourth stage is XA PREPARE. In fact, the data has been transferred to the slave database through Binlog. Even if the main database is hung up and the transaction is still in the XA PREPARE state after reconstruction, we can think that the data will not be lost. The last stage is XA COMMIT.

RadonDB divides these five stages into three stages: begin, execute and commit.

RadonDB implements distributed transactions at the SI isolation level. There is a Commit Lock in Radon, and this level of isolation cannot be achieved without this lock. So what is the SI isolation level? SI is an acronym for SNAPSHOT ISOLATION, which is invisible to uncommitted transactions. For example, when there are three partitions, when none of them have XA commit, the data of uncommitted transactions cannot be seen when other transactions read. Another role is to partially commit invisible, or there are three partitions, the first partition XA commit, the other two partitions are preparing commit, at this time if there are other clients to read the data is not visible.

In order to detect the isolation level of XA, we have developed an open source tool, which is relatively simple: one update thread keeps updating, and 16 table scanning threads keep scanning the table. If the distributed transaction does not meet the SI isolation level, it is possible for 16 table sweep threads to see part of the update thread's data.

We have conducted more than 10 billion operations and tests, and the process is random. We will shut down the master node of the storage layer to do the "master-slave" switch. In a large number of tests, it has not been found that part of the data has been read.

Let's take a look at another component of RadonDB-Xenon, a highly available MySQL management tool. Suppose a node has one master, two slaves and three MySQL, how to achieve the high availability between them?

The working mechanism of Xenon is to cooperate with MySQL, keep going to ping MySQL through MySQL links, and get MySQL information. A MySQL corresponds to a Xenon and is deployed in a container, with one master and two slaves distributed in different container. When the Master is unavailable, other Xenon will not detect the heartbeat sent by the Master, then the heartbeat initiated by the Xenon will initiate the selection master operation, and other slave nodes will be selected as the new master node.

Next, let's talk about how Xenon initiates the selection master operation, how to select a new master node, and how to ensure that data is not lost after the selection.

There are several challenges for a "master" multi-slave MySQL cluster to achieve high availability: the first is how to choose "master"; the second is how to synchronize data with the original Master after selecting "master" to ensure that data is not lost; the third is how to choose "master" as soon as possible, and how to apply data and provide services to the new master node as soon as possible after the original "master" dies.

We combined the GTID of MySQL with the choice of Raft. Xenon mainly implements the main function of Raft selection, combined with MySQL GTID to achieve high availability. Friends who know the Raft algorithm may know that Raft mainly does two things, the first one is to choose "master". The second is log synchronization. Xenon selects "master" and uses Raft to select the master protocol. After selecting "master", it will be combined with MySQL GTID for data synchronization.

If it is a "master" and "two slaves" node, which of the two slave nodes will be selected as the new master? This is a combination of MySQL's GTID and semi-sync. When we set the vote_timeout of semi-sync to infinitely long, we can basically think of it as "master". After writing, at least one "slave" will be received and returned to the "master", and the "master" will be returned to cluster to ensure that at least one "slave" and "master" data are fully synchronized. When the "master" is dead, the slave node which is completely synchronized with the master node data will be selected as the new master node, and then quickly replay and provide services according to the parallel replication of MySQL.

After introducing Radon and Xenon, let's take a look at how the data is distributed in RadonDB? the underlying storage of RadonDB is based on MySQL, that is, one master and two slaves managed by Xenon is a node, and the whole storage layer is composed of multiple such nodes.

Users need to specify a partitioning key when creating a table, and RadonDB divides the entire large table into 32 small tables according to the partitioning key. The allocation rule is that the whole table has 4096 slots, of which each small table has 32 slots.

The smallest unit of RadonDB is a small table, named T1140000 to T1160031. Each small table occupies 128 slots. For example, the first small table is from 0 to 127. In this way, when users are doing Insert, they can judge which small table the data will fall into.

How does RadonDB expand its capacity? the smallest unit of RadonDB is a small table, but the numbers 4096 and 128are configurable. In terms of capacity expansion, RadonDB allows small watches to drift dynamically between different machines. Because it is a MySQL table, it is easy to float a small table from one MySQL instance to another MySQL instance. The first is to do a full migration, write down the location of the migration at that time, and then add to the increment. This method of migration based on small tables not only does not affect the read and write operation, but also is convenient to operate, which can not only expand the capacity, but also reduce the capacity.

RadonDB also supports Binlog. Why? Because RadonDB is a distributed database, if there are other databases or data that want to subscribe to RadonDB data, you can subscribe to RadonDB Binlog. After connecting to the RadonDB, execute SHOW BINLOG EVENTS to specify which GTID to start with, and you can also specify how many items to subscribe to. In this way, RadonDB data can be imported into heterogeneous databases in real time.

If RadonDB receives more complex AP operations, such as JOIN, what is its mechanism? RadonDB also has a computing node. When the user SQL comes up, RadonDB will automatically route the request to the compute node if it finds that there are more complex AP operations such as JOIN.

The computing node is plug-in, it can be other powerful AP analytical database, after the computing node calculates the results, RadonDB will automatically feedback to the client, in this case, the client can not be aware of these operations.

The advantage of this is that transactional and computational are resource-isolated, but the disadvantage is that two copies of storage are required. How to overcome the shortcomings? In fact, at present, we do not have a very good way, only through compression to solve this problem temporarily.

RadonDB also implements the audit log function. When the user's request reaches RadonDB, RadonDB records the user's request to the local disk. We can audit through multiple dimensions in the figure above, and we can also query slow operations. When the number of log requests is large, RadonDB will clean it up regularly. At the same time, RadonDB also supports a variety of audit modes, such as read-only audit, write-only audit, read-write audit and so on.

You may be worried that it will be difficult to export a large amount of data in a distributed database. In response to this situation, RadonDB provides import and export tools that are parallel and faster than MySQL's native Mydumper.

RadonDB provides full-link monitoring, and if the distributed database is a black box, it is not easy to troubleshoot problems. RadonDB does three layers of monitoring from the back, the first layer is show processlist, this layer is to monitor users' connection to RadonDB, which is the same as MySQL. RadonDB implements an ordinal list, and the purpose of this layer is to see the SQL statements executed from cluster to RadonDB. The second layer is to monitor which stage of the peak transaction within the RadonDB is executed, which can be monitored through the show txnz command; the last layer is show queryz, which can see which backend the specific clause is executed.

Through these three layers of monitoring, you can quickly locate specific problems, such as a slow MySQL, where it is slow.

The above figure is a performance comparison table, the above RadonDB is four storage nodes, and the following is a stand-alone MySQL. As you can see, the performance of the RadonDB is basically three times that of the stand-alone, while the latency is basically 1 Unix 3. Why did this happen? This is the power of distribution. Suppose we have a 1TB table, and if we use a stand-alone database, the Btree will be higher and the IO depth path for each request will be longer. RadonDB divides the 1TB data into four nodes, assuming that each node is 250G on average, and each node is subdivided into each small table. When a user requests, we only need to request a small table, and RadonDB executes all requests in parallel, depending on the slowest small table. So in this design, the performance of the RadonDB will basically be three times that of the stand-alone, and the latency will be 1amp 3.

Finally, let's talk about the prospect of RadonDB. We all know that NewSQL like Google Spanner will be a major trend, and many companies are completely developing NewSQL on their own. Some people think that the traditional way of dividing libraries and tables based on MySQL is out of date, but we put forward a new concept-MyNewSQL, which is the combination of MySQL and NewSQL.

In fact, RadonDB is a MyNewSQL, which brings the commonly used algorithms in the field of NewSQL to MySQL, thus realizing MyNewSQL. The function of RadonDB is basically the same as that of NewSQL, but it is stored based on MySQL, the table and data structure can be heterogeneous, and the performance is greatly improved.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.