Shulou(Shulou.com)06/01 Report--

This article shows you how to understand the persistence of troubleshooting errors and the use of checkpoint. The content is concise and easy to understand, which will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

In terms of CheckPoint, sometimes it will have an effect on the failure, when is it most effective? If caching BlockManager will save the data to memory or disk according to your caching strategy! Persistence, most of the time, will work properly. But I am afraid that sometimes there will be accidents. For example, data cached in memory may be inexplicably lost. The Executor process is down. In other words, the data stored in the disk file is inexplicably gone, and the file is deleted by mistake. Although the enterprise has never encountered it, it is possible.

When this happens, next, if you want to do something on this RDD, you may find that one of the partition of RDD can't be found. Recalculate the missing partition, and then cache and use it after calculation. Sometimes, calculating a RDD can be extremely time-consuming. Maybe there were a large number of parent RDD before RDD. So if you want to recalculate a partition, you may have to recalculate the partition of all previous parent RDD.

In this case, you can choose to checkpoint the RDD, just in case. Checkpoint, that is, persists a copy of RDD data to a fault-tolerant file system (such as hdfs). When calculating this RDD, if you find that its cached data is missing. The priority is to find out if there is any checkpoint data (go to hdfs). If any, checkpoint data is used. Not so much as to recalculate.

Checkpoint, in fact, can be used as a spare for cache. If cache fails, checkpoint can be used. Checkpoint has both advantages and disadvantages, the advantage is that it improves the reliability of spark operations, once problems occur, it is still very reliable, there is no need to recalculate a large number of rdd;, but the disadvantage is that when performing checkpoint operations, that is, when rdd data is written into hdfs, it will consume performance.

Checkpoint, trade performance for reliability. First do the buffer and then do the checkpoint, for example, to the HDFS, that is, from the cache to the HDFS above the checkpoint operation! Later, we will use this RDD, and then it actually has a component called CacheManager, and then CacheManager will go to the BlockManager to find the data. If there is any, we will get the data through CacheManager. If not, we will get the data from checkpoint.

Checkpoint principle:

In the code, with SparkContext, set up a checkpoint directory, which can be a directory of a fault-tolerant file system, such as hdfs

In the code, execute RDD.checkpoint () on the rdd that needs to be checkpoint

RDDCheckpointData (API within spark), which takes over your RDD, will be marked as marked for checkpoint, ready for checkpoint

After your job has finished running, a finalRDD.doCheckpoint () method will be called. It will scan backwards along the rdd lineage and find that there is a rdd marked as waiting for checkpoint. It will be marked twice, inProgressCheckpoint, and is accepting the checkpoint operation.

After the job is executed, a new internal job is launched to write all the data of the rdd marked as inProgressCheckpoint to the hdfs file. (note: if the rdd has been cache before, the data will be fetched directly from the cache and written to the hdfs; if there is no cache pass, the rdd will be recalculated and then checkpoint)

Change the dependency rdd before the rdd of checkpoint to a lineage that CheckpointRDD*, forces to change your rdd. Later, if the cache data acquisition of rdd fails, it will directly obtain the checkpoint data through its upstream CheckpointRDD, to the fault-tolerant file system, such as hdfs.

Talk about the use of checkpoint

SparkContext, set the checkpoint directory

Checkpoint the RDD

The above is how to understand the persistence of troubleshooting errors and the use of checkpoint. Have you learned any knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.