Shulou(Shulou.com)06/02 Report--

This article introduces you what are the relevant parameters of Fair Scheduler, the content is very detailed, interested friends can refer to, hope to be helpful to you.

First, in yarn-site.xml, set the configuration parameter yarn.resourcemanager.scheduler.class to org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.

The configuration option of Fair Scheduler consists of two parts, one of which is in yarn-site.xml, which is mainly used to configure parameters at the scheduler level, and the other is in a custom configuration file (default is fair-scheduler.xml), which is mainly used to configure information such as resource volume and weight of each queue.

To understand what Fair Scheduler is, read my article, "Hadoop Fair Scheduler Analysis."

1. Profile yarn-site.xml

(1) yarn.scheduler.fair.allocation.file: customize the location of the XML configuration file, which is mainly used to describe the attributes of each queue, such as resource quantity, weight, and so on. The specific configuration format will be described later.

(2) yarn.scheduler.fair.user-as-default-queue: whether to specify a user name as the queue name of the application when the application does not specify a queue name. If set to false or not, all applications with unknown queues will be submitted to the default queue, with the default value of true.

(3) yarn.scheduler.fair.preemption: whether to enable preemption mechanism. The default is false.

(4) yarn.scheduler.fair.sizebasedweight: when allocating resources within a queue, by default, fair polling is used to allocate resources to each application. This parameter provides another resource allocation method: resources are allocated according to the number of application resource requirements, that is, the more resources are required, the more resources are allocated. By default, this parameter value is false.

(5) yarn.scheduler.assignmultiple: whether to start the batch allocation function. When there are a large number of resources in a node, it can be allocated at one time or multiple times. By default, this parameter value is false.

(6) yarn.scheduler.fair.max.assign: if batch allocation is enabled, you can specify the number of container allocated at a time. By default, the parameter value is-1, which means there is no limit.

(7) yarn.scheduler.fair.locality.threshold.node: a skipped resource scheduling opportunity that an application can accept when it requests resources on a node. When resources on one node can be assigned to an application according to the allocation policy, if the node is not the node expected by the application, you can choose to skip the allocation opportunity to temporarily allocate resources to other applications until there are node resources that meet the needs of the application. Typically, a heartbeat represents a scheduling opportunity, and this parameter represents the proportion of skipping scheduling opportunities to the total number of nodes, which, by default, is-1.0, indicating that no scheduling opportunities are skipped.

(8) yarn.scheduler.fair.locality.threshold.rack: a resource scheduling opportunity that can be skipped when an application requests resources on a rack.

(9) yarn.scheduler.increment-allocation-mb: memory normalization unit, default is 1024, which means that if a Container request resource is 1.5GB, it will be normalized by the scheduler to ceiling (1.5GB / 1GB) * 1G=2GB.

(10) yarn.scheduler.increment-allocation-vcores: virtual CPU normalization unit. The default is 1, which is similar to memory normalization unit.

two。 Custom Profil

Fair Scheduler allows users to specifically place queue information in a profile (default is fair-scheduler.xml). For each queue, the administrator can configure the following options:

(1) minResources: the minimum resource guarantee is set to "X mb, Y vcores". When the minimum resource guarantee of a queue is not satisfied, it will obtain resources first than other peer queues. For different scheduling policies (described in more detail later), the meaning of the minimum resource guarantee is different. For fair policy, only memory resources are considered. That is, if a queue uses more memory resources than its minimum resources, it is considered to be satisfied. For the drf policy, consider the amount of resources used by the main resource, that is, if the amount of the main resource of a queue exceeds its minimum, it is considered to be satisfied.

(2) maxResources: the maximum amount of resources that can be used. Fair scheduler ensures that the amount of resources used by each queue will not exceed the maximum available resources of the queue.

(3) maxRunningApps: the maximum number of applications running at the same time. By limiting this number, the intermediate output produced when the excess Map Task is running at the same time can be prevented from exploding the disk.

(4) minSharePreemptionTimeout: the minimum shared quantity preempts the time. If the amount of resources used by a resource pool during that time has been lower than the minimum amount of resources, it begins to preempt resources.

(5) schedulingMode/schedulingPolicy: the scheduling mode of the queue, which can be fifo, fair or drf.

(6) aclSubmitApps: a list of Linux users or user groups that can submit the application to the queue, which is "*" by default, indicating that any user can submit the application to the queue. It is important to note that this property is inherited, that is, the list of child queues inherits the list of parent queues. When you configure this property, you use "," to split between users or user groups, and spaces between users and user groups, such as "user1, user2 group1,group2".

(7) aclAdministerApps: list of administrators for this queue. The administrator of a queue can manage the resources and applications in the queue, such as killing any application.

Administrators can also add maxRunningJobs attributes to individual users to limit the maximum number of applications they can run at the same time. In addition, the administrator can set the default values for the above properties with the following parameters:

(1) userMaxJobsDefault: the default value of the user's maxRunningJobs attribute.

(2) defaultMinSharePreemptionTimeout: the default value of the minSharePreemptionTimeout attribute of the queue.

(3) defaultPoolSchedulingMode: the default value of the schedulingMode attribute of the queue.

(4) fairSharePreemptionTimeout: fair sharing quantity preempts time. If the amount of resources used by a resource pool during that time has been less than half of the amount of fair sharing, it begins to preempt resources.

[example] suppose you want to set three queues queueA, queueB and queueC for a Hadoop cluster, where queueB and queueC are sub-queues of queueA, and it is stipulated that ordinary users can run up to 40 applications at the same time, but user userA can run up to 400 applications at the same time, then the following settings can be made in the custom configuration file:

100 mb, 100 vcores 150 mb, 150 vcores 200 300 1.0 30 mb, 30 vcores 50 mb, 50 vcores 400 40 6000 about Fair Scheduler related parameters are shared here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.