Shulou(Shulou.com)05/31 Report--

This article introduces the relevant knowledge of "how to see the Map process of RDD through map operation". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

How are the operations such as map,flatMap in RDD strung together to form a DAG diagram? This is a very important question, and if you understand it, you can better understand the kernel implementation of Spark. This article attempts to explain this through the map process.

Let's first take a look at a subclass of RDD: MapPartitionsRDD, which will be used in the map function scenario.

Its definition is:

Private [spark] class MapPartitionsRDD [U: ClassTag, T: ClassTag] (var prev: RDD [T], f: (TaskContext, Int, Iterator [T]) = > Iterator [U], / / (TaskContext, partition index, iterator) preservesPartitioning: Boolean = false, isFromBarrier: Boolean = false, isOrderSensitive: Boolean = false) extends RDD [U] (prev)

Prev is the parent RDD, which is the input parameter of the parent class RDD, which is firstParent in the following code.

F represents the definition of the map function, where the second Int parameter is the partition index number. Regardless of how the f input parameter came in, let's take a look at what MapPartitionsRDD needs to do.

As mentioned earlier, the most important function for RDD is the compute method definition of compute,MapPartitionsRDD:

Override def compute (split: Partition, context: TaskContext): Iterator [U] = f (context, split.index, firstParent [T] .iterator (split, context))

It is clear that the f function of the input parameter is executed with the current solit partition!

So, how did this MapPartitionsRDD come into being? It was originally generated by the map function in the RDD class:

Def map [U: ClassTag] (f: t = > U): RDD [U] = withScope {val cleanF = sc.clean (f) new MapPartitionsRDD [U, T] (this, (_, iter) = > iter.map (cleanF))}

What do these lines of code mean? It still needs a good analysis here.

Compared with the definition of MapPartitionsRDD, we know:

(_, iter) = > iter.map (cleanF)

The _, _ represents TaskContext and partition index, because there are split input parameters and context input parameters in the compute method of MapPartitionsRDD, so there is no need to pass these two parameters in RDD.

Iter represents the dataset to be processed and is defined in the compute method in MapPartitionsRDD as:

FirstParent [T] .iterator (split, context)

The function is the dataset of the split partition of the first parent class RDD. It's clear here to cleanF this dataset (that is, the map function after sc.clean, sc.clean means to remove the bytecode that cannot be serialized and distribute it to other nodes for execution).

This is the end of the content of "how to see the Map process of RDD through map operation". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.