Shulou(Shulou.com)05/31 Report--

This article mainly explains "how to understand the persistence of RDD". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "how to understand the persistence of RDD".

One of the most important functions of Spark is to persist (or cache) a dataset in memory between different operations. When you persist a RDD, each node stores its chunked results in memory and reuses them in other actions on the dataset (or derived dataset). This will make the subsequent action (Actions) faster (usually 10 times faster). Caching is the key to building iterative algorithms with Spark.

You can mark a RDD to be persisted with the persist () or cache () method, and then once the calculation is triggered by an action (Action) for the first time, it will be retained in the memory of the computing node and reused. Cache has a fault-tolerant mechanism so that if any partition of RDD is lost, it will be automatically recalculated by using the transformation operation that originally created it (not all recalculations, only the missing parts). When you need to delete the persisted RDD, you can use unpersistRDD () to do the job.

In addition, each RDD can be saved at a different save level, allowing you to persist the dataset on your hard disk, or as a serialized Java object in memory (saving space), or even copy across nodes. These rank selections are determined by passing an org.apache.spark.storage.StorageLevel object to the persist () method. The cache () method is a shortcut that uses the default storage level, that is, StorageLevel.MEMORY_ONLY (to store deserialized objects in memory).

StorageLevel has five attributes, namely: whether useDisk_ uses disk, whether useMemory_ uses memory, whether useOffHeap_ uses out-of-heap memory, such as whether Tachyon,deserialized_ is deserialized, and the number of replication_ backups.

The complete optional storage levels are as follows:

Selection of storage level

The different storage levels of Spark are designed to meet different requirements in memory usage and CPU efficiency tradeoffs. We recommend that you choose through the following steps:

If your RDDs fits well with the default storage level (MEMORY_ONLY), you don't need to make any changes. This is already the most efficient option for CPU, and it makes RDDs operation as fast as possible. If not, try using MEMORY_ONLY_SER and choose a fast serialization library so that objects can still be accessed quickly even with high space usage.

Do not store it on the hard disk as much as possible, unless the functions that calculate the dataset have a large amount of computation, or they are filtered

A lot of data. Otherwise, recalculating a partition is almost as fast as reading from a hard disk.

Thank you for your reading, the above is the content of "how to understand the persistence of RDD". After the study of this article, I believe you have a deeper understanding of how to understand the persistence of RDD, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.