Shulou(Shulou.com)06/01 Report--

This article mainly introduces "MySql's Binlog log tool example analysis". In daily operation, I believe many people have doubts in MySql's Binlog log tool example analysis. The editor consulted all kinds of data and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts of "MySql Binlog log tool example analysis". Next, please follow the editor to study!

Canal

Location: based on database incremental log parsing, provide incremental data subscription-consumption, currently mainly supports mysql.

Principle:

Canal simulates the interaction protocol of mysql slave, disguises itself as mysql slave, and sends dump protocol to mysql master

Mysql master receives the dump request and starts pushing binary log to slave (that is, canal)

Canal parses binary log objects (originally byte streams)

The whole parser process can be roughly divided into several steps:

Connection gets the location of the last successful parsing (if started for the first time, the initial location or the binlog site of the current database)

Connection establishes a connection, BINLOG_DUMP command occurs

Mysql starts pushing Binary Log

The received Binary Log parses the protocol through Binlog parser to add some specific information.

Passing it to the EventSink module for data storage is a blocking operation until the storage is successful

After the storage is successful, record the Binary Log location regularly.

Data filtering: supports wildcard filtering mode, table name, field content, etc.

Data routing / distribution: solve 1 parser (1 store corresponding to multiple store mode)

Data merging: resolve n 1 (multiple parser corresponds to 1 store)

Data processing: perform additional processing before entering store, such as join

Maxwellcanal is developed by Java, divided into server and client, has a large number of derivative applications, stable performance, powerful; canal needs to write its own client to consume the data parsed by canal. The advantage of maxwell over canal is that it is easy to use, it directly outputs data changes to json strings, and there is no need to write clients.

DatabusDatabus is a low-latency change capture system, which has become an indispensable part of LinkedIn data processing pipeline. Databus addresses the basic requirements of reliably capturing, flowing, and handling major data changes. Databus provides the following features:

Isolation between source and consumer

Ensure high availability in sequence and at least one delivery

Consumption starts at any point in time in the change stream, including the full boot function of the entire data.

Regional consumption

Source consistency preservation

At this point, the study on "MySql's Binlog log tool example analysis" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.