Shulou(Shulou.com)05/31 Report--

This article introduces the knowledge of "how to design the architecture of distributed MySQL Binlog storage system". Many people will encounter this dilemma in the operation of actual cases, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

1. Introduction to kingbus 1.1.What is kingbus?

Kingbus is a distributed MySQL binlog storage system based on raft strong consistency protocol. It can act as a MySQL Slave to synchronize binglog from a real Master and store it in a distributed cluster. At the same time, it acts as a MySQL Master to synchronize the binlog in the cluster to other Slave. Kingbus has the following characteristics:

Compatible with MySQL replication protocol, synchronize binlog on Master by Gtid, and support slave to pull binlog from kingbus by Gtid.

Cross-region data replication. Kingbus supports cross-region data replication through raft protocol. The binlog data written to the cluster is strongly consistent across multiple nodes and the binlog order is exactly the same as on the master.

High availability, because kingbus is built on the Raft strong consistent protocol, which can achieve the survival of more than half of the nodes in the cluster, the entire binlog pull and push service is highly available.

1.2 what problem can kingbus solve?

Kingbus can reduce the network traffic of Master. In an one-master-multi-slave replication topology, Master needs to send binlog to each slave. If there is too much slave, the network traffic is likely to reach the upper limit of the network card of Master. For example, performing operations such as delete big table or online DDL in Master may cause a large number of binlog event to be generated instantly. If 10 slave are hung under the master, the traffic of the network card on the master will be magnified 10 times. If master uses a gigabit network card, it is possible to fill its network card if it generates traffic above 10MB/S. Through the way that kingbus is connected to master, slave can be distributed to multiple machines to balance the traffic.

To simplify the Master Failover process, simply upgrade a Slave connected to the kingbus to Master, and repoint the kingbus to the new Master, while the other slave is still connected to the kingbus, and the replication topology remains the same.

Save space for Master to store binlog files. Generally speaking, there are more expensive SSD on the MySQL. If the binlog file takes up more space, the data stored in MySQL has to be reduced. You can reduce the amount of storage of binlog files on Master by storing all the binlog in kingbus

Heterogeneous replication is supported. Through Alibaba's open source canal connection to kingbus,kingbus, continue to push binlog to canal,canal to receive binlog and then push to kafka message queue, and finally store it in HBase. The business unit uses Hive to write SQL directly to achieve real-time business analysis.

Overall architecture of 2.kingbus

The overall architecture of kingbus is shown in the following figure:

Storage is responsible for storing raft log entry and Metadata. In kingbus, raft log and mysql binlog are fused together. By distinguishing different header information, the data part of raft log is binlog event, so there is no need to store the two types of log separately and save storage space. Because kingbus needs to store some meta-information, such as raft node voting information, the specific content of some special binlog event (FORMAT_DESCRIPTION_EVENT).

Raft replicates the Lead election, log replication and other functions of the kingbus cluster, using etcd raft library.

Binlog syncer, which runs only on the Lead node of the Raft cluster, has only one syncer for the entire cluster. Syncer disguises as a slave and establishes a master-slave replication connection to the Master. Master will filter the binlog event that the syncer has accepted according to the executed_gtid_set sent by the syncer, and only send the binlog event that the syncer has not received. This set of replication protocol is fully compatible with the MySQL master-slave replication mechanism. After syncer receives the binlog event, it does some processing according to the binlog event type, and then encapsulates the binlog event into a message and submits it to the raft cluster. Through the raft algorithm, the binlog event can be stored in multiple nodes and achieve strong consistency.

Binlog server is a Master that implements the replication protocol. The real slave can be connected to the port where binlog server listens. Binlog server will send binlog event to slave. The whole process of sending binlog event is implemented according to MySQL replication protocol. When no binlog event is sent to slave, binlog server sends heartbeat event to slave periodically to keep the replication connection alive.

Api server, which is responsible for the management of the entire kingbus cluster, including the following:

Raft cluster membership operation, check cluster status, add a node, remove a node, update node information, etc.

Binlog syncer related operations, start a binlog syncer, stop the binlog syncer, and check the binlog syncer status.

Binlog server related operations, start a binlog server, stop the binlog server, and check the binlog server status. All kinds of exceptions in the server layer do not affect the raft layer, and server can be understood as a plug-in that starts and stops on demand. When you extend kingbus later, you only need to implement the server of the relevant logic. For example, if you implement a server of the kafka protocol, you can consume messages in kingbus through kafka client.

Core implementation of 3.kingbus 3.1Core implementation of storage

There are two log forms in storage, one is the raft log (hereinafter called raft log), which is generated and used by the raft algorithm, and the other is the user form of Log (that is, mysql binlog event). In the design of Storage, two Log forms are combined into a Log Entry. It's just distinguished by different head information. Storage consists of a data file and an index file, as shown in the following figure:

Segment is of a fixed size (1GB) and can only be appended. The name is first_raft_index-last_raft_index, which indicates the raft index range of the segment.

Only one segment is writable, its file name is first_raft_index-inprogress, and the other segment is read-only.

Both the read-only segment and the corresponding index file are written and read through mmap.

* the index content of a segment is stored on both disk and memory. Reading the index only needs to be read in memory.

3.2 use of the etcd raft library

Etcd raft library is single-threaded when dealing with Apply logs, committed entries, and so on. Refer to the link for the specific function. The processing time of this function should be as short as possible. If the processing time exceeds the raft election time, it will cause the cluster to be re-elected. This requires special attention.

Core implementation of 3. 3 binlog syncer

The main tasks of binlog syncer are as follows:

Pull binlog event

Parse and process binlog event

Submit the binlog event to the raft cluster. It is obvious that the processing speed of the whole process can be raised through the pipeline mechanism, and the kingbus of each stage is handled by a separate goroutine, and the different stages are connected through pipes. Because binlog syncer is received one by one according to binlog event, syncer does not guarantee transaction integrity, so you may need to reconnect Master after syncer fails. At this time, * A transaction may be incomplete and binlog syncer needs the ability to discover transaction integrity. Kingbus implements the function of transaction integrity resolution, which is fully implemented by referring to MySQL source code.

3.4 Core implementation of binlog server

Binlog server implements the function of a master. When slave establishes a replication connection with binlog server, slave sends relevant commands, and binlog server needs to respond to these commands. Finally send binlog event to slave. For each slave,binlog server, a goroutine is started to read the raft log continuously, and the relevant header information is removed, which becomes binlog event, and then sent to the slave.

This is the end of the content of "how to design the architecture of distributed MySQL Binlog storage system". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.