Shulou(Shulou.com)05/31 Report--

This article mainly introduces "what is the count operation of the spark kernel RDD". In the daily operation, I believe that many people have doubts about the count operation of the spark kernel RDD. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubt of "what is the count operation of the spark kernel RDD?" Next, please follow the editor to study!

Count method of RDD source code:

As you can see from the above code, the count method triggers a call to the runJob method of SparkContext:

Further trace the runJob (rdd, (context: TaskContext, iter: Iterator [T]) = > func (iter), partitions, allowLocal) method:

2. Clean (func):

Code analysis:

Enter submitJob (rdd, func, partitions, callSite, allowLocal, resultHandler, properties):

The above code analysis:

3.1.1. Enter the new JobWaiter (this, jobId, partitions.size, resultHandler) method

We can see that we are sending messages to ourselves.

3.1.3. Enter the dagScheduler.handleJobSubmitted (jobId, rdd, func, partitions, allowLocal, callSite,listener, properties) method

First build finalStage, and then another getMissingParentsStages method. You can find that there are two modes of running: local running and cluster running. Local running is mainly used for local experiments and debugging:

3.1.3.1. Enter the finalStage = newStage (finalRDD, partitions.size, None, jobId, callSite) method:

3.1.3.2. Enter the runLocally (job) method:

3.1.3.3.2. Enter the getMissingParentStages (stage) .sortBy (_ .id) method:

Follow up the getShuffleMapState method:

3.1.3.3.3. Enter the submitMissingTasks (stage, jobId.get) method:

At this point, the study on "what is the count operation of spark kernel RDD" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.