Shulou(Shulou.com)05/31 Report--

This article introduces the knowledge of "how spark uses the stack". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

The spark process runs as a JVM process, and the stack size can be configured with-Xmx and-Xms. How does it use the stack? The following is the spark memory allocation diagram.

Storage memory

The default JVM heap for spark is 512MB, and only 90% is used to avoid OOM errors. Set through spark.storage.safetyFraction. Spark stores the data that needs to be processed through memory, uses 60% of the safe space, and is controlled by spark.storage.memoryFraction. If we want to know how much data can be cached by spark? Assuming that the number of executors used is N, then the cached data is N*90%*60%*512MB. # # the memory of shuffle memory shuffle memory is "Heap Size" * spark.shuffle.safetyFraction * spark.shuffle.memoryFraction. The default spark.shuffle.safetyFraction is 0.8 and spark.shuffle.memoryFraction is 0.2, so the memory with shuffle memory 0.8*0.2*512MB = 0.16*512MB # # unroll memory unroll memory is spark.storage.unrollFraction * spark.storage.memoryFraction * spark.storage.safetyFraction, that is, 0.2 * 0.6 * 0.9 * 512MB = 0.108 * 512MB. Unroll memory is used for data serialization and deserialization.

That's all for "how spark uses the stack". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.