Shulou(Shulou.com)06/03 Report--

Tuesday, 2019-2-19

Yarn task scheduling-capacity scheduler (capacity scheduling) / fair scheduler (fair scheduling)

Due to the limited cluster resources, when many application resource requests cannot be satisfied, yarn needs appropriate policies to schedule application resource requests.

Overview of Scheduler

There are three scheduling strategies implemented in Yarn: FIFO/Capacity/Fair Schedulers

(1) FIFO Scheduler:

Queue all application in the order in which they are submitted, first in first out

Advantages-> easy to understand and without any configuration

Disadvantages-> applications that are not suitable for shared clusters; will fill up the cluster resources and cause a large number of applications to wait.

Summary:

1. A queue can use all the resources of yarn

2. Later submitted tasks must wait for the previous tasks to be run before resources can be obtained and executed.

(2) Capacity Scheduler / / see the reference link in this section in detail

The application is divided into multiple task queues, and each queue has corresponding resources inside the queue. Resource allocation follows the FIFO policy. Queue resources support flexible adjustment: the free resources of one queue can be allocated to the "Hunger" queue. (note: once the demand for the previous idle queue grows, because preemption is not supported and the kill resource container cannot be forced, you need to wait for other queues to release resources. To prevent this from happening, you can configure the maximum resource of the queue to limit)

Task queue supports inheritance structure

Summary:

Maximize cluster throughput

Core thought

-Cluster resources are shared by multiple queues

-idle queues can "lend" resources to busy queues

-you can take it back when you need it.

Scheduling strategy

-Resources that should be acquired / resources actually acquired, select the queue with the lowest ratio

-FIFO in queue

-consider limitations: individual users use resources, use other queue resources

(3) Fair Scheduler

There is no need to reserve resources for a specific small application, but to make a dynamic fair allocation when it needs to be executed; dynamic resource allocation has a delay because it needs to wait for large job to release part of the resources Small job resources are used, large job can get all the resources again Fair Scheduler also supports scheduling between application queue

Summary:

Multi-user fair sharing of cluster resources

Job pool

-each user has a separate resource pool

-put jobs in the shared resource pool

-minimum resource guarantee for each job

Scheduling strategy

-default FIFO

-the scheduling policy in the queue is configurable

/ / the design goal of the Fair scheduler is to allocate fair resources to all applications (the definition of fairness can be set by parameters). The Yarn scheduler comparison diagram above shows fair scheduling of two applications in a queue; of course, fair scheduling can also work between multiple queues. For example, suppose there are two users An and B, each with a queue. When A starts a job and B has no tasks, A will get all the cluster resources; when B starts a job, A's job will continue to run, but after a while the two tasks will each get half of the cluster resources. If B starts the second job at this time and the other job is still running, it will share the resources of the queue B with the first job of B, that is, the two job of B will be used for 1/4 of the cluster resources, while the job of A will still be used for half of the cluster resources, and the result is that the resources will eventually be shared equally between the two users.

Reference link:

Http://www.mamicode.com/info-detail-1097801.html

Three schedulers of Hadoop: FIFO, Capacity Scheduler, Fair Scheduler:

Https://blog.csdn.net/xiaomage510/article/details/82500067

Comparison between Fair Scheduler and Capacity Scheduler

Https://blog.51cto.com/qishi23/1599138

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.