Spark tuning (3): RDD refactoring and persistence

First, the reconstruction and optimization of RDD architecture

Try to reuse RDD, similar RDD, can be extracted as a common RDD for later RDD calculation, repeated use.

Second, public RDD must be persistent.

For public RDD that needs to be calculated and used multiple times, be sure to persist.

Persistence, that is, caching RDD data to memory / disk, (BlockManager). No matter how many times the RDD is calculated later, the persistent data of the RDD will be taken directly, for example, a piece of data will be extracted directly from memory or disk.

Third, persistence can be serialized.

If data is normally persisted in memory, it may result in excessive memory footprint, which may lead to an OOM memory overflow.

When pure memory cannot support the full storage of public RDD data, it is preferred to use serialization to store it in pure memory. Serialize the data of each partition of RDD into a large array of bytes, just one object; after serialization, the memory footprint is greatly reduced.

The only drawback to serialization is that it needs to be deserialized when getting data.

If serialization of pure memory mode still results in OOM, memory overflow; you can only consider disk mode, memory + disk normal way (no serialization).