Shulou(Shulou.com)05/31 Report--

This article mainly explains "what are MySQL Binlog processing tools". The explanation in this article is simple and clear, easy to learn and understand. Please follow the ideas of Xiaobian and go deep into it slowly to study and learn "MySQL Binlog processing tools" together!

Canal

Positioning: Based on database incremental log parsing, providing incremental data subscription & consumption, currently mainly supporting mysql.

Principle:

canal emulates the interactive protocol of mysql slave, disguises itself as mysql slave, sends dump protocol to mysql master mysql master receives dump request, starts pushing binary log to slave(i.e. canal)canal parses binary log object (originally byte stream)

The whole parser process can be roughly divided into several steps:

Connection Gets the location of the last successful resolution (If it is started for the first time, obtain the initial specified position or the binlog position of the current database) Connection establishes a connection, and BINLOG_DUMP command Mysql starts to push Binary Log received. Binary Log is parsed through Binlog parser, and some specific information is supplemented and passed to EventSink module for data storage. It is a blocking operation until the storage is successful. After the storage is successful, the Binary Log position is recorded regularly. Data filtering: Support wildcard filtering mode, table name, field content, etc. Data routing/distribution: solve 1:n (one parser corresponds to multiple stores) Data merging: solve n:1 (multiple parsers correspond to one store) Data processing: perform additional processing before entering the store, such as joinMaxwell

Canal is developed by Java, divided into server and client, has a large number of derivative applications, stable performance, powerful;canal needs to write its own client to consume the data parsed by canal.

Maxwell's advantage over canal is that it is simple to use, it directly outputs data changes as a json string, and there is no need to write the client.

Databus

Databus is a low-latency change capture system that has become an integral part of LinkedIn's data processing pipeline. Databus addresses the basic requirements for reliably capturing, flowing and handling major data changes. Databus provides the following features:

Isolation between source and consumer guarantees sequential and at least one-time delivery with high availability Consumption starts at any point in the change stream, including full boot functionality for the entire data. Partition consumption source consistency preservation Aliyun data transmission service DTS

Data Transmission Service (DTS) is a data flow service provided by Alibaba Cloud that supports data interaction between RDBMS (relational database), NoSQL, OLAP and other data sources. DTS provides multiple data transmission capabilities such as data migration, real-time data subscription, and real-time data synchronization. It can implement multiple business application scenarios such as non-stop data migration, data remote disaster recovery, remote multi-activity (unitization), cross-border data synchronization, real-time data warehouse, query report diversion, cache update, asynchronous message notification, etc., helping you build a highly secure, scalable, and highly available data architecture.

Advantages: Data Transmission Services DTS supports data transmission between RDBMS, NoSQL, OLAP and other data sources. It provides data migration, real-time data subscription and real-time data synchronization and other data transmission methods. Compared with third-party data flow tools, DTS provides more diverse, high-performance, high-security and reliable transmission links, and it provides many convenient functions, which greatly facilitates the creation and management of transmission links.

Personal understanding: is a message queue, will push it to you wrapped sql objects, you can do a service to parse these sql objects.

Avoid costly deployment and maintenance costs. DTS is adapted to Aliyun RDS (online relational database), DRDS and other products, solving the subscription high availability problem in Binlog log recovery, active/standby handover, VPC network handover and other scenarios. At the same time, targeted performance optimizations were performed for RDS. For stability, performance and cost considerations, it is recommended.

Thank you for reading, the above is "MySQL Binlog processing tools what" content, after the study of this article, I believe we have a deeper understanding of MySQL Binlog processing tools what this problem, the specific use of the situation also needs to be verified. Here is, Xiaobian will push more articles related to knowledge points for everyone, welcome to pay attention!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.