Shulou(Shulou.com)06/01 Report--

This article mainly introduces "what is the difference between Flink's checkpoint and savepoint". In daily operation, I believe many people have doubts about the difference between Flink's checkpoint and savepoint. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful to answer the question of "what is the difference between Flink's checkpoint and savepoint?" Next, please follow the editor to study!

1. Checkpoint

Checkpoint is used to recover from checkpoint when an flink application fails. For example, when an application crashes when an exception occurs, JVM does not shut down. Instead, it attempts to restart from a checkpoint. You can customize the restart policy of the application. When the restart fails, and you want to run again from the state saved by the checkpoint, execute

Flink run-s hdfs://master:8020/flink/checkpoint03/s1/savepoint-0b3f0b-ed13f369aadc-c flink.ceshi / opt/flink_path/sbt-solr-assembly.jar

Different state backend, its operator state and checkpoint storage location are different.

2. Savepoint

Savepoint (SavePoint): it is a special checkpoint, but unlike checkpoint, which is triggered periodically from the system, it is triggered by the user through commands. The storage format is also different from checkpoint. The data will be stored in a standard format. No matter what the configuration is, Flink will recover from this checkpoint.

Savepoint triggers Checkpoint by manual commands and persists the results to the specified storage path. Its main purpose is to help users save the state data in the system during the process of upgrading and maintaining the cluster, so as to avoid the situation that the system can not return to the original computing state due to the normal termination of application operations such as downtime operation and maintenance or upgrade applications, thus unable to achieve end-to-end Exactly-Once semantic guarantee.

1) configure the storage path of Savepoints

Configure the location of the SavePoint storage in flink-conf.yaml, and after setting up, if you want to create a SavePoint of the specified Job, you do not have to specify the location of the SavePoint when you execute the command manually.

State.savepoints.dir: hdfs:/hadoop101:9000/savepoints

2) set the operator ID in the code

In order to be able to upgrade smoothly between different versions of jobs and between different versions of Flink, it is highly recommended to manually assign ID to operators, which will be used to determine the state range of each operator. If you do not manually assign ID to each operator, Flink will automatically generate an ID for each operator, and these automatically generated ID depend on the structure of the program and are sensitive to code changes. Therefore, it is strongly recommended that you set up ID manually.

Stream.flatMap (_ .split (")) .uid (" flapMap-001 ") / / each operator specifies a uid to facilitate the recovery of the operator's state from the SavePoint .map ((_, 1)) .uid (" map=001 ") .keyby (0) .sum (1) .uid (" sum-001 ") .print ()

3) trigger SavePoint

/ / start SavePoint [root @ hadoop101 bin] #. / flink run-c com.bjsxt.flink.state.TestSavepoints-d / home/Flink-Demo-1.0-SNAPSHOT.jar// manually trigger Job.Then cancel the job [root@hadoop101 bin] #. / flink savepoint 6ecb8cfda5a5200016ca6b01260b94ce [root] #. / flink cancel 6ecb8cfda5a5200016ca6b01260b94ce

4) restore jobs from SavePoint

Flink run\-s hdfs://hadoop101:9000/savepoints/savepoint-6ecb8c-e56ccb88576a\-c com.bjsxt.flink.state.TestSavepoints\-d / home/Flink-Demo-1.0-SNAPSHOT.jar at this point, the study on "what is the difference between Flink's checkpoint and savepoint" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.