Shulou(Shulou.com)06/01 Report--

How to understand spark on yarn memory allocation, I believe that many inexperienced people do not know what to do, so this paper summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

1. Basic concepts to master this knowledge point, to understand the following points:

How on-heap memory for spark driver and executor is configured. How off-heap memory for spark driver and excutor is configured. The minimum scheduling unit of yarn. First, let's talk about the above three concepts: 1). The default value of spark.driver.memory is 1g. The on-heap memory of the driver process, and the driver process is the process in which the sparkcontext is initialized. The heap memory of driver in client mode is not set by SparkConf, but replaced with the-- driver-memory command, or configured in the default configuration file. 2)。 The spark.driver.memoryOverhead default value is max (DriverMemory*0.1384m). In the cluster mode of YARN or kubernetes mode, the total amount of off-heap memory requested by the driver is usually 6% of the driver heap memory. 3)。 The default value of spark.executor.memory is 1g. Total jvm memory of the Executor. 4)。 Spark.executor.memoryOverhead defaults to max (executor*0.1384). The total amount of off-heap memory requested by a single executor. This parameter can only be used on yarn or kubernetes, usually 0. 06-0. 1 of executor memory. 5) .spark.yarn.am.memory defaults to 512m memory of MagneAppmaster in client mode. If it is Cluster mode, spark.driver.memory will be used instead. 6) .Yarn 's minimum scheduling unit yarn scheduling container has a minimum granularity, which must be an integral multiple of the resource scheduling. The minimum granularity configuration parameter is that the yarn.scheduler.minimum-allocation-mb minimum scheduling unit default value is 1024, and the unit is mb. 2.cluster mode the case here is 40 executor, a single executor memory request is 20GB executor driver is 20GB web ui screenshot of yarn: the total memory requested by the whole APP is 923648MB=902GB. The single Executor memory requested by spark app can be seen from 1.3,1.4:

20GB * 0.1 + 20GB=22GB, then 40 executor memories are:

In the cluster mode of 22GB * 40 = 880GBspark on yarn, the Driver side is with appmaster, so the appmaster memory parameters are invalid, so the total memory of driver and executor is available from 1.1 and 1.2: 20GB * 0.1 + 20GB=22G, then the calculated total memory is: 22GB+88 GB happens to be 902GB. In the case calculated above, since the minimum granularity of yarn cluster memory scheduling here is 1GB, there is no participation in the above calculation. In 3.cluster mode

At this point, it is assumed that there are 10 executor, and the memory of each executor is 19GB. The memory of the driver is still 19GB. First, let's take a look at the total memory of web ui:

Converted to GB, it is 236544MB/1024 = 231GB.

Then the calculated memory that can guide a single executor is: the minimum scheduling unit of 19*1.1GB = 20.9GBYarn is 1GB, so when a single executor is scheduled, the memory is 21GB. The total memory of 10 executor is 21 * 10 = 210GB. After calculation, the memory of driver is 19 million 1.1 GB = 21GB, then the calculated spark app memory is the same as the memory on yarn ui. 4. Pay attention

For the above calculation to be correct, it is necessary to ensure that dynamic executor application and release is not enabled. If it is enabled, it should be calculated on a case-by-case basis. After reading the above, have you mastered how to understand spark on yarn memory allocation? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.