Shulou(Shulou.com)06/01 Report--

This article mainly introduces "what is Spark's failover fault-tolerant mechanism". In daily operation, I believe many people have doubts about what Spark's failover fault-tolerant mechanism is. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the doubts about "what is Spark's failover fault-tolerant mechanism?" Next, please follow the editor to study!

The SPARK computing framework implements the overall failover mechanism in three ways:

1. The checkpoint on the driver side is implemented in the Driver layer, which is used to restore the Driver site after the Driver crash.

(note that this checkpoint and RDD's checkpoint are two different things.)

2. The replication on the executor is used in Receiver to solve the problem of the loss of unsaved data after a single executor dies. 3. WAL: implemented in Driver and Receiver, used to solve: (1) if Driver hangs, all executor will hang up, then all unsaved data will be lost, and replication will no longer work; (2) after Driver hangs, which block is registered in driver before hanging up, and which block is assigned to the running batch job before hanging up, this information is lost. So you need WAL to persist this information. (the question of task assignment) if you don't think what you said is enough, you can draw a picture and continue to talk about it.

The blue arrow indicates that the data is read and stored in executor memory. If WAL is enabled, the data will be written to the log file where there is a fault-tolerant file system (wal on the executor side).

The cyan arrow reminds driver that the metadata of the received data block is sent to the SparkContext in driver, including the reference ID of the data block in executor memory and the offset information of the data block in the log file (wal on the driver side)

The yellow arrow indicates checkpoint these calculations, which are used to reproduce the driver scene. Streaming will be periodically checkpoint to the file (checkpoint on the driver side)

Speaking of which, congratulations, you can confuse the interviewer.

What are the five features of Spark's RDD? This question can make a lot of people confused, in fact, the statement of these five features is a comment in the source code of Spark, as follows:

Let me explain briefly:

1. Partition

Partition is the basic unit of a dataset, and each shard is processed by a computing task, which determines the granularity of parallel computing. The number of slices defaults to the number of core.

The storage of each shard is implemented by BlockManager, and each partition is logically mapped to a Block of BlockManager, and this Block is calculated by a Task. 2. Partitioner a partitioner, that is, the sharding function of RDD. Currently, two types of sharding functions are implemented in Spark, one is hash-based HashPartitioner and the other is range-based RangePartitioner. Only for RDD of key-value will there be Partitioner. The Partitioner function determines not only the number of slices for RDD itself, but also the number of slices for parent RDD Shuffle output. 3. RDD in compute funcSpark is calculated in shards, and each RDD implements the compute function to achieve this purpose. The compute function composes iterators, eliminating the need to save the results of each calculation. 4. Each transformation of dependencyRDD generates a new RDD, so there is a pipelined forward-backward dependency between RDD.

When some partition data is lost, Spark can recalculate the lost partition data through this dependency instead of recalculating all partitions of RDD. 5. PreferredLocation a list that stores the priority location (preferred location) for each Partition. For a HDFS file, this list holds the location of each Partition block. According to the concept of "mobile data is not as good as mobile computing", Spark assigns computing tasks to the storage location of the data blocks it wants to process as much as possible when scheduling tasks. At this point, the study on "what is the failover fault-tolerant mechanism of Spark" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.