Shulou(Shulou.com)06/01 Report--

This article introduces the principle and management of Yarn, the content is very detailed, interested friends can refer to, hope to be helpful to you.

The CGroup scheme is used for the management of CPU in Yarn, even if CGroup is used to manage the computing power of Container in Yarn. The scheduler in RM allocates the corresponding amount of Container of resources according to the resource request of AM, which mainly includes memory and CPU. The process of CPU allocation is the same as that of memory allocation. The editor focuses on the management of CPU in Container.

Yarn contains two types of container execution scripts. If you want to control CPU, you need to use LinuxContainerExecutor.java to start the container. The default DefaultContainerExecutor.java cannot manage CPU. You can set the yarn.nodemanager.Linux-Container-executor.resources-handler.class parameter value in the configuration file to org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler.

LinuxContainerExecutor can run container as the user to which the container belongs, while the default DefaultContainerExecutor can only run container as a normal administrator.

First of all, we need to pay attention to the two parameters in the configuration file. The path to the configuration file in the source code is hadoop-yarn-api\ src\ main\ java\ org\ apache\ hadoop\ yarn\ conf\ YarnConfiguration.java

The first parameter represents the number of CPU cores on this node that can be allocated to the container. It is indicated by vcores. By default, virtual CPU is used in 8 CPU yarn to eliminate the difference between different CPU performance. By default, one physical CPU corresponds to a virtual CPU, which needs to be set manually.

The second parameter represents the percentage of usage time that the CPU on the node can be allocated to all containers. The default is 100%, that is, all the computing power of CPU is used by Yarn.

Another parameter is used to limit whether excess computing power can be allocated to other processes when the CPU is idle. By default, if set to TRUE, the container's CPU usage time is strictly controlled according to the resource limit. This parameter is as follows:

After understanding a few main parameters, let's take a look at how the CPU usage of the container is controlled.

The control logic code is in the file hadoop-yarn-server\ hadoop-yarn-server-nodemanager\ src\ main\ java\ org\ apache\ hadoop\ yarn\ server\ nodemanager\ util\ CgroupsLCEResourcesHandler.java.

Take a look at a few key logical points:

These two methods are called before the container starts and after the container executes. The specific calling process is called in the launchContainer method in hadoop-yarn-server\ hadoop-yarn-server-nodemanager\ src\ main\ java\ org\ apache\ hadoop\ server\ nodemanager\ LinuxContainerExecutor.java.

The first method is mainly used to set the CPU usage limit of the container, which is implemented in the setupLimits method:

This method first gets the number of virtual CPU allocated in the Container, and then creates the file path of the CGroup, that is, the path associated with the CPU and container name in the CGroup path. In this method, you can see that there are two ways to restrict CPU:

The first is through the cpuShares parameter, which calculates the usage time based on the number of virtual CPU allocated in the container. For example, if container An is 1 vcores and container B is 2 vcores, the CPU quota that containers An and B can use at full load is 1:2. This ratio is relative, that is, if there is only one container, 100% CPU time can be used. This approach can make more full use of resources while ensuring fairness.

The second method is to enable the cgroups.strict-resource-usage parameter mentioned above, that is, set it to TRUE, get the specific limit by calling the getOverallLimits method, and then write the limit to the cfs_period_us and cfs_quota_us files. This method limits the absolute usage time. Take a look at the getOverallLimits method:

Several parameters are involved here:

YarnProcessors: represents the number of virtual CPU that can be used in yarn on a node, and the minimum cannot be less than 0.01.The value of this parameter is initialized by calling the NodeManagerHardwareUtils.getContainersCores method in the init method, that is, the number of CPU available to Yarn is calculated according to the CPU allocation ratio set in the configuration file:

QuotaUS: the cpu time that can be used in a time period.

PeriodUS: time period (per cpu core).

The specific calculation method is in the getOverallLimits method:

The default value of quotaUS is 1000 * 1000, that is, 1 second, and the default value of quotaUS is (1000 * 1000/yarnProcessors) rounded.

When yarnProcessors is less than 1, the value of quotaUS is (periodUS * yarnProcessors) rounded, but the minimum value that cannot be less than 1000ms is 1000 * 1000, that is, 1 second.

These two values are returned and written to the file.

When the container finishes executing, it cleans up the CPU quota configuration set above, which is implemented in the following method:

About Yarn implementation principles and management is how to share here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.