Shulou(Shulou.com)06/03 Report--

Fair Scheduler summary

The default fair scheduling strategy is only based on memory resources, and task scheduling based on memory and CPU resources can be configured by using DRF (Dominant Resource Fairness).

When only one application is running, the application can use the entire cluster. When other applications are submitted to the cluster, the idle resources will be allocated to the newly submitted applications, so that each application can get roughly the same resources. The Hadoop default scheduler forms a queue of applications so that small applications can be completed within a reasonable time frame without keeping large applications waiting for resources for a long time. This is also a reasonable way to share cluster resources among many users. Finally, fair scheduling can set the priority of the application, and set the priority weight to determine the proportion of resources occupied by the application.

Hierarchical queues that can be plugged into policies

Fair scheduling supports hierarchical queues, and all queues extend from root queues. A typical fair scheduling method is to distribute available resources among the subqueues of available root queues. The application can only be scheduled in the leaf queue. One queue can be designated as a subqueue of another queue. Specifying a hierarchical queue requires that the parent queue be specified at the same time. For example, root.queue1 root.queue1.queue2.

Fair scheduling can be configured with custom scheduling policies, such as FifiPolicy,FairSharePolicy,DominantResourceFairnessPolicy

Fair scheduling requires the following to be configured in yarn-site.xml

Yarn.resourcemanager.scheduler.class org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler

Fair scheduling configuration mainly consists of two files, the first is yarn-site.xml, under the Hadoop configuration directory, and then you need to create a configuration file fair-scheduler.xml, which mainly configures the weight of the queue and resource capacity.

10000 mb,0vcores 90000 mb,0vcores 50 0.1 2.0 fair charlie 5000 mb,0vcores 0.5 40000 mb,0vcores 3.0 30 5

Queue access control list

ACLs controls queues and manages which users can access specific queues. It mainly includes aclSubmitApps and aclAdministerApps configuration items, which can be configured for each queue.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.