Shulou(Shulou.com)06/02 Report--

Today, I will talk to you about the scheduling and isolation of memory and CPU resources in Hadoop YARN. Many people may not know much about it. In order to let you know more, Xiaobian summarizes the following contents for you. I hope you can gain something according to this article.

Hadoop YARN supports scheduling of both memory and CPU resources (only memory is supported by default, if you want to further schedule CPU, you need to make some configuration yourself). Xiaobian will introduce how YARN schedules and isolates these resources.

In YARN, resource management is done by both ResourceManager and NodeManager, where the scheduler in ResourceManager is responsible for resource allocation and NodeManager is responsible for resource provisioning and isolation. After the ResourceManager allocates resources on a NodeManager to tasks (this is called "resource scheduling"), the NodeManager needs to provide corresponding resources for tasks according to requirements, and even ensure that these resources should be exclusive, providing basic guarantees for task operation. This is called resource isolation.

For a detailed introduction to Hadoop YARN Resource Scheduler, please refer to my article: YARN/MRv2 Resource Manager Deep Dive-Resource Scheduler.

Before we get into the details of resource scheduling and isolation, let's take a look at the characteristics of memory and CPU, two different resources. The amount of memory resources will determine the life and death of the task. If there is not enough memory, the task may fail. In contrast, CPU resources will only determine the speed at which the task runs, and will not have an impact on life and death.

[Scheduling and isolation of memory resources in YARN]

Based on the above considerations, YARN allows users to configure the physical memory resources available on each node. Note that this is "available" because the memory on a node will be shared by several services, such as part for YARN, part for HDFS, part for HBase, etc. YARN configures only what it can use. The configuration parameters are as follows:

（1）yarn.nodemanager.resource.memory-mb

Indicates the total amount of physical memory available to YARN on the node. The default is 8192 MB. Note that if your node memory resources are not enough 8GB, you need to reduce this value. YARN will not intelligently detect the total amount of physical memory on the node.

（2）yarn.nodemanager.vmem-pmem-ratio

For every 1MB of physical memory used by a task, the maximum amount of virtual memory that can be used is 2.1 by default.

（3） yarn.nodemanager.pmem-check-enabled

Whether to start a thread to check the amount of physical memory being used by each task. If the task exceeds the allocated value, kill it directly. Default is true.

（4） yarn.nodemanager.vmem-check-enabled

Whether to start a thread to check the amount of virtual memory being used by each task. If the task exceeds the allocated value, kill it directly. Default is true.

（5）yarn.scheduler.minimum-allocation-mb

The minimum amount of physical memory that a single task can request is 1024 (MB) by default. If the amount of physical memory requested by a task is less than this value, the corresponding value is changed to this number.

（6）yarn.scheduler.maximum-allocation-mb

The maximum amount of physical memory that a single task can request. The default is 8192 MB.

By default, YARN uses thread monitoring to determine whether a task is overusing memory, and if it finds an overuse, it will directly kill it. Because Cgroups lack flexibility in memory control (i.e., tasks cannot exceed the memory limit at any time, and if they exceed it, they will be killed directly or OOM will be reported), while Java processes will double the memory at the moment of creation and then plummet to normal values. In this case, thread monitoring is more flexible (when it is found that the process tree memory doubles instantaneously beyond the set value, it can be considered normal and will not kill the task), so YARN does not provide Cgroups memory isolation mechanism.

Scheduling and Isolation of CPU Resources in YARN

In YARN, the way CPU resources are organized is still being explored, and at present (version 2.2.0) is only a preliminary, very coarse-grained implementation, and a more fine-grained CPU partition has been proposed and is being refined and implemented.

CPU is divided into virtual CPU (CPU virtual Core), where the virtual CPU is YARN itself introduced the concept, the original intention is, considering that different nodes CPU performance may be different, each CPU has different computing power, such as a physical CPU computing power may be twice that of another physical CPU, at this time, you can make up for this difference by configuring several virtual CPUs for *** physical CPU. When a user submits a job, he or she can specify the number of virtual CPUs required for each task. In YARN, CPU-related configuration parameters are as follows:

（1）yarn.nodemanager.resource.cpu-vcores

Indicates the number of virtual CPUs available to YARN on this node. The default value is 8. Note that it is recommended to set this value to the same value as the number of physical CPU cores. If your node has less than 8 CPU cores, you need to reduce this value, and YARN will not intelligently detect the total number of physical CPUs in the node.

（2） yarn.scheduler.minimum-allocation-vcores

The minimum number of virtual CPUs that a single task can request. The default value is 1. If the number of CPUs requested by a task is less than this number, the corresponding value is changed to this number.

（3）yarn.scheduler.maximum-allocation-vcores

The maximum number of virtual CPUs that can be requested for a single task. The default is 32.

By default, YARN does not schedule CPU resources. You need to configure the corresponding resource scheduler for you to support. For details, please refer to my two articles:

(1) Hadoop YARN configuration parameter analysis (4)-Fair Scheduler related parameters

(2) Hadoop YARN configuration parameter analysis (5)-Capacity Scheduler related parameters

By default, NodeManager doesn't do any isolation of CPU resources, and you can enable CPU isolation by enabling Cgroups.

Due to the uniqueness of CPU resources, this CPU allocation method is still coarse-grained. For example, many tasks may be IO intensive and consume very little CPU resources. If you allocate a CPU for it at this time, it is a serious waste. You can let him share a CPU with several other tasks. That is, we need to support a more granular CPU expression.

Drawing lessons from the division of CPU resources in Amazon EC2, it is proposed that the minimum unit of CPU is EC2 Compute Unit (ECU), and one ECU represents the processing power equivalent to 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor. YARN proposes the minimum CPU unit YARN Compute Unit (YCU), currently an integer, the default is 720, set by the parameter yarn.nodemanager.resource.cpu-ycus-per- core, indicating the computing power of a CPU core (This feature does not exist in version 2.2.0, and may be added to version 2.3.0). In this way, when the user submits the job, he can directly specify the required YCU. For example, the specified value is 360, which means that 1/2 CPU cores are used. In fact, only 1/2 of the calculation time of one CPU core is used. Note that at the operating system level CPU resources are allocated by time slice, you could say that a process uses 1/3 of CPU time slice, or 1/5 of CPU time slice.

At present, YARN memory resource scheduling draws lessons from the way in Hadoop 1.0, which is more reasonable, but the CPU resource scheduling method is still being improved. At present, it is only a preliminary rough implementation. I believe that in the near future, the CPU resource scheduling in YARN will be more perfect.

After reading the above, do you have any further understanding of how memory and CPU resources are scheduled and isolated in Hadoop YARN? If you still want to know more knowledge or related content, please pay attention to the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.