Shulou(Shulou.com)05/31 Report--

This article mainly introduces "what RDD is provided by Spark". In daily operation, I believe many people have doubts about which RDD problems are provided by Spark. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful for you to answer the questions of "what RDD does Spark provide?" Next, please follow the editor to study!

Go deep into RDD

RDD itself is an abstract class with many concrete implementation subclasses:

RDD is calculated based on Partition:

The default Partitioner is as follows:

The documentation of HashPartitioner is as follows:

Another common Partitioner is RangePartitioner:

RDD needs to consider the memory strategy for persistence:

Spark offers a number of StorageLevel options:

At the same time, Spark provides unpersistRDD:

There is also a very important CheckPoint operation for RDD itself:

The details of doCheckpoint are as follows:

Take NewHadoopRDD as an example, the internal information is as follows:

Take WholeTextFileRDD as an example, the internal information is as follows:

When RDD generates a job call, the classic process is as follows:

At this point, the study of "which RDD is provided by Spark" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.