Shulou(Shulou.com)06/02 Report--

Today, I will talk to you about how to handle exceptions in Debezium. Many people may not know much about it. In order to make you understand better, the editor summed up the following content for you. I hope you can get something from this article.

Debezium is an open source CDC tool developed based on Confluent's Connect platform. Debezium is a distributed CDC tool that integrates a variety of databases and is designed not to lose any events. Of course, when the system configuration is normal, the operation is normal, and the management is correct, Debezium can provide database event sending for exactly once. Of course, if you encounter some system problems, you will not lose any data, although it may produce several duplicate pieces of data during the recovery process. Therefore, in some abnormal cases, Debezium and kafka provide at least once event sending.

1. Configuration and startup errors

Configuration and startup exception

The Connector component will fail at startup, then print the error/exception log in the log, and stop connector if some parameters are abnormal and the mysql cannot be connected properly. Or when mysql restarts, reading the binlog log has been lost from history, and the last binlog location cannot be read.

In this case, these exceptions provide some hints for action. Connector restarts after configuration correction and mysql issues are resolved.

2.MySQL becomes unavailable

When mysql is dead,

Once Connector runs and mysql becomes unavailable for some reason, connetor stops and exits. Just restart connector when mysql is restored.

Note that when using GTID in a mysql cluster highly available environment, you can restart connector directly. Connector automatically rushes to the binlog of each machine from the mysql server cluster, and finds the location where the binglog was last read, and continues to read.

If connector does not use GTID, then the binlog record of connector will only remember the binlog location of the mysql machine to which it is connected. So if you want to repair, also need to repair the corresponding mysql server in order to provide the correct service.

3.Kafka Connect process stops gracefully

Kafka connection processing crashes

If Kafka Connect is running under a distributed model, then Kafka Connect suddenly stops abnormally. Then Kafka Connect will migrate all the process groups running on the current machine to other machines before shutdown. This results in a short delay in data processing.

4.Kafka Connect process crashes

If the kafka Connect crashes, then the other Connector will restart execution from the previously saved point. In distributed mode, other machines start running automatically.

By default, Connector starts reading from the last offset, and this time it produces some duplicate data, and the amount of duplicate data is based on the gap between the last submitted job.

PS. The event captured by Debezium is idempotent, so even if there is duplicate data, only one state is guaranteed.

5.Kafka becomes unavailable

Since all Connector have the opportunity to Kafka, the producer API,Kafka Connect of kafka periodically records the latest offset, according to the configuration of kafka Connect.

If Kafka Broker is down, kafka Connector will periodically try to connect to Kafka brokers.

In other words, if kafka connector recovers when kafka broker reruns.

6. Connector is stopped for a duration

Connector stopped for a while.

By default, Connector restart allows you to continue to connect to the mysql and start reading data from the original position.

If you stop for too long and mysql has deleted the binlog, you need to configure it manually or start the snapshot mode.

After reading the above, do you have any further understanding of how to handle exceptions in Debezium? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.