Shulou(Shulou.com)05/31 Report--

This article introduces the relevant knowledge of "what are the core operations of RDD in Spark". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

What exactly is RDD in Q1:Spark?

RDD is the core abstraction of Spark, and RDD can be thought of as a "distributed functional programming language".

RDD has the following core characteristics:

A list of partitions

A function for computing each split

A list of dependencies on other RDDs

Optionally, a Partitioner for key-value RDDs (e.g. To say that the RDD is hash-partitioned)

Optionally, a list of preferred locations to compute each split on (e.g. Block locations for an HDFS file)

There are two core operations in RDD: Transformation and Action,Transformation only record metadata for data operations, and Action calculates the data and produces results.

What type of RDD are Q2:Checkpoint and persist?

The Operation of RDD is divided into two categories, transformation and action, in which transformation generates new RDD,action and generates new data. The lineage as DAG stores the transformation, executes lineage when action and generates data.

Checkpoint and persist are two special operations of RDD. Persist persists RDD, and checkpoint persists RDD while cutting off historical lineage.

Persist and checkpoint violate the operation of immutability. They actually modify the storage level and lineage in RDD meta info and return the modified RDD object itself instead of the new RDD object.

Where does the Driver program run when Q3:Spark is running?

In Standalone mode, Driver runs on the client that submits the Spark Application

The client is able to submit Spark programs because Spark is installed

Driver is responsible for running the program.

Q4: what is the secret of understanding DAGScheduler's division of DAG Stage?

Generally speaking, reading data from the outside, performing Shuffle operations and writing data will become the boundary of Stage division.

The internal operation of Stage is Pipeline, which can greatly improve the efficiency of the program.

Shuffle is the dividing point of two Stage

Q5: how to understand Narrow Dependencies and Wide Dependencies?

Narrow Dependencies and Wide Dependencies constitute Spark Lineage

Narrow Dependencies: for example, map, filter, union, join with inputs co-partitioned

Wide Dependencies: for example, groupByKey, join with inputs not co-partitioned

The key to judging that it is Narrow Dependencies is that the result of the left RDD Partition operation is the only RDD Partition on the right.

The key to judging Wide Dependencies is that the result of the left RDD Partition operation is at least two RDD Partitions on the right.

This is the end of the content of "what are the core operations of RDD in Spark". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.