How to use checkpoint for failure recovery in Structured Streaming

What this article shares with you is about how Structured Streaming uses checkpoint for fault recovery. The editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

Using checkpoint for fault recovery

In the event of a failure or shutdown, you can restore the progress and status of the previous query and continue execution from where it was stopped. This is done using Checkpoint and pre-written logging. You can configure the query with the checkpoint location, and the query saves all progress information (that is, the offset range processed in each trigger) and run aggregations (for example, wordcount in the example) to the checkpoint location. This checkpoint location must be a path in the HDFS-compatible file system and can be set as an option in DataStreamWriter when the query is started.

AggDF

.writeStream

.outputMode ("complete")

.option ("checkpointLocation", "path/to/HDFS/dir")

.format ("memory")

.start ()

The specific test code is as follows:

Val lines = spark.readStream.format ("socket"). Option ("host", "localhost"). Option ("port", 9999). Load ()

Val words = lines.as [string] .flatMap (_ .split (""))

Val wordCounts = words.groupBy ("value") .count

Val query = wordCounts.writeStream.queryName ("aggregates"). OutputMode ("complete"). Option ("checkpointLocation", "memory/"). Format ("memory"). Start ()

Spark.sql ("select * from aggregates". Show ()

Kill dropped submit to test.

The above is how Structured Streaming uses checkpoint for fault recovery. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.