Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "what is the Apache Mesos scheduling mechanism". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

1. Preface

Scheduler is the core component of Mesos, which is mainly responsible for allocating resources on each slave to each framework. Common scheduling mechanisms are FIFO,Fair Scheduler,Capacity Scheduler,Quincy,Condor and so on. In order to support multi-framework access, Mesos adopts a two-layer scheduling mechanism. First, the allocator in mesos allocates resources to framework, and then framework's own scheduler allocates resources to tasks.

2. Mesos scheduling mechanism

The scheduling mechanism in Mesos is called "Resource Offer", which adopts a resource-based scheduling mechanism, which is different from the slot-based mechanism in Hadoop. In mesos, slave reports the amount of resources (CPU and memory) directly to master, and master allocates the amount of resources to framework according to some mechanism, of which "some mechanism" is "Dominant Resource Fairness (DRF)".

For a system like mesos with a two-tier scheduling framework, the following problems need to be solved in the design: "how can Mesos meet the requirements of each framework without knowing its resource requirements?" More specifically, "how does Mesos do data locality when it doesn't know which data is stored in framework and which nodes?" To solve this problem, mesos provides a "reject offer" mechanism that allows framework to temporarily reject slave that does not meet its resource requirements. Here, mesos uses a "delay scheduling" scheduling mechanism similar to that in Hadoop.

In mesos, job scheduling is a distributed process, when failure occurs, it needs to show a certain degree of efficiency and robustness. To do this, mesos provides the following mechanisms:

(1) filters mechanism. During each scheduling process, mesos-master needs to communicate with framework-scheduler. If some framework always reject slave, then the scheduling performance will be inefficient because of the extra communication overhead. To avoid unnecessary communication, mesos provides a filters mechanism that allows framework to receive only "slave with more than L remaining resources" or "only slave in the node list".

(2) rescinds mechanism. If a framework does not return the corresponding task for the assigned resource within a certain period of time, the mesos reclaims the amount of its resource and allocates those resources to another framework.

3. Dominant Resource Fairness (DRF)

DRF is a max-min fair resource allocation mechanism that supports multiple resources, where max represents max {CPU,mem} and min represents min {user1,user2, … } = min {max {CPU1,mem1}, max {CPU2,mem2}, … }, where user represents framework in mesos, and the pseudo code of the algorithm is shown below:

For example, suppose there are 9 CPUs and 18 GB RAM in the system, and two user (framework) run two kinds of tasks, respectively, and the amount of resources required is and. For user A, each task consumes 1 task 9 of the total CPU and 2 picks 9 of the total memory, so the dominant resource of An is memory; for user B, each task consumes 1 task 3 of the total CPU and 1x18 of the total memory, so the dominant resource of B is CPU. DRF will balance the dominant resources of all users, that is, A can run 3 task; and B can run 2 task. In this way, each user gets the same proportion of dominant resources, that is, A gets 2task; 3's RAMs,B and 2 3's CPUs.

A possible scheduling sequence for the DRF algorithm is shown in the following figure:

The advantage of DRF is that it can satisfy four features, namely: Sharing incentive,Strategy-proofness,Envy-freeness and Pareto efficiency

4. Mesos scheduling problem

The DRF scheduling algorithm in Mesos pursues too much fairness and does not take into account the actual application needs. In the actual production line, we often need a scheduling mechanism similar to the Capacity Scheduler in Hadoop, which divides all resources into several queue, each queue allocates a certain amount of resources, and each user has a certain upper limit of resource usage; the more used scheduling strategy is that each queue can customize its own scheduler strategy, such as FIFO,Priority and so on.

Because Mesos uses a two-tier scheduling mechanism, it will face the problem of design decision in actual scheduling: which scheduling mechanisms will be implemented by * * layer and layer 2 schedulers respectively, that is, will most of the scheduling mechanisms be put into the * layer scheduler, or will the * layer scheduler only support simple resource allocation (the allocation ratio is specified by the administrator)?

Mesos adopts the Resource Offer mechanism (different from the slot-based scheduling mechanism in Hadoop). This scheduling mechanism faces the problem of resource fragmentation, that is, all the resources on each node cannot be allocated, and the remaining points may not be enough for any task to run, which is similar to the memory fragmentation problem in the operating system.

This is the end of the content of "what is the Apache Mesos scheduling mechanism". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.