Shulou(Shulou.com)06/01 Report--

How to analyze the implementation of Mesos resource scheduler, for this problem, this article describes the corresponding analysis and solution in detail, hoping to help more small partners who want to solve this problem find a simpler and easier way.

1 mesos DRF Introduction

The mesos scheduler is implemented according to the DRF (dominant resource fairness) algorithm. The intuitive idea behind the DRF algorithm is that in a multi-resource environment, a user's resource allocation should be determined by the user's dominant share, which is the resource that occupies the largest share of all the multiple resources that have been allocated to the user.

Mesos uses resource offers to allocate resources between frameworks. A Resource offer is a set of free resources on multiple slave nodes. The Master determines how many resources to provide to each framework according to the scheduling policy by sending to the framework in the form of a Resource offer, and then the framework responds to the Resource offer, acknowledging the used resources in the Resource offer and returning the remaining free resources.

Mesos resource allocator is an implementation of hierarchical weighted max-min fairness. By abstracting a role concept, the framework is grouped according to roles. Then resource allocation can be divided into two layers: first, ranking among framework groups by weighted DRF algorithm; second, ranking each framework by weighted DRF algorithm within framework groups. Then, according to the final sorting result, Resource Offer is made for each framework from small to large. This is shown below.

Implementation of 2 mesos resource allocator

Mesos scheduler implementation is divided into two parts, Allocators and sorters, Allocator defines and implements the resource allocator interface and logic. Sorter sorts resource consumers using a specific resource allocation algorithm.

The implementation of Sorter is DRFSorter, which uses the DRF algorithm to rank resource consumers.

Allocator implementation HierarchicalAllocatorProcess implements a hierarchical allocator that groups the framework, sorting between and within groups using DRFSorter.

DRFSorter

DRFSorter sorts clients by the dominant share value of each user. Several concepts in DRFSorter are as follows:

Client: The user of the resource. defined as follows

struct Client

{

std::string name; //resource consumer name

double share; //share of resources allocated to resource users

};

The sorting between clients is to compare the size of the share value first, and if the share is equal, then the size of the name needs to be compared.

Resource: Represents a resource that contains the resource name, the resource value, and the framework preferences for the resource. Pb is defined as follows:

message Resource {

required string name = 1; //resource name

required Value.Type = 2; //Resource Value Type Scalar

optional Value.Scalar scalar = 3; //Value type is scalar, both CPU and memory value types are scalar

optional Value.Ranges ranges = 4; //Value types are ranges

optional Value.Set set = 5; //Value type is set

optional string role = 6 [default = "*"]; //framework preferences for resources

}

Mesos application environment is a cluster environment of multiple resource types, and all resources that can be applied for by the framework are represented by Resource array.

Weight: The weight of resource allocation for each client. Type defined as double

As shown in the figure below, in order to sort the clients, DRFSorter needs to maintain some information internally, as shown in the figure below:

dirty: indicates that the resource allocation of DRFSorter has changed, and the share value of each client needs to be recalculated.

clients: A collection of resource consumers, using a red-black tree, sorted by the dominant share value of the client.

Resources: The total resource value owned by this DRFSorter.

There are two types of interfaces for the entire DRFSorter. The first type is to add, delete, and modify clients and resources. When resources change, DRFSorter will set dirty to true. The second category sorts clients. If dirty is true, the dominant share value will be recalculated for all clients, and then reinserted into the collection to return the sorted result.

The core sorting process of DRFSorter is as follows:

| If the dirty flag bit is true

| Traversing the clients collection, calculating the share value of each client

| Calculate the ratio of the value of each resource of client to the total value of DRFSorter resources of this type.

Find the maximum value and divide by the client weight to get the client share value.

The formula is: Share =

, the value of resource i assigned to client,

is the total value of resource i in DRFSorter. W is the weight of the client.

| Insert client again and reorder.

| Returns sorted results.

Allocator

A hierarchical allocator is implemented, as shown in the following figure. Users can group frameworks according to roles, specify the weight of framework groups in RoleInfo, and have a DRFSorter to sort the resource allocation of each framework group. Then, there is a DRFSorter within each framework group to sort the resource allocation of each framework within the group.

Some of the Allocator concepts are as follows

Framework: Application framework, obtains cluster resources from mesos, issues specific computing tasks, and is the consumer of the cluster.

struct Framework

{

hashset filters; //framework filters

bool checkpoint; //Is checkpoint in progress

FrameworkInfo info; //framework info

};

Slave: background tasks running on each cluster node, performing specific computing tasks and reporting node resources and loads.

struct Slave

{

.............

Resources available; //currently available resources

bool whitelisted; //Whether it is on the whitelist. If false, resource offer cannot be made.

bool checkpoint; //Whether checkpoint is in progress, if checkpoint is in progress, resource offer cannot be made

SlaveInfo info; //slave info

};

Role: Used to group frameworks, you can specify a weight for each group of frameworks. Role information RoleInfo is as follows.

message RoleInfo {

required string name = 1;

optional double weight = 2 [default = 1];

}

Whitelist: Specify a valid slave, if a whitelist is made, then the slave in the whitelist is valid.

Filters: Used by the framework to filter resources from slaves, and can be used to reject resources on specific slaves.

In order to allocate resources, the allocator needs to maintain the following information internally, as shown in the figure below.

frameworks: Mapping of frameworks, from framework Id to framework information.

slaves: mapping of slaves, mapping from slave Id to slave information.

roles: Mapping of framework grouping information.

FrameworkSorters: Mapping of DRF sorting containers within a framework group.

Role Sorter: Container for DRF sorting between groups of frames.

Allocator core scheduling logic:

| Sort framework groups from small to large (framework is grouped by role)

| Traversing the sorted framework group

| Sort frameworks within a group

| Traversing the sorted framework

| Traversing the slave collection, sending all available slave resources that meet the requirements to the framework

| Extract all available resources whose role is "*" in slave,"*" indicates general universal resources.

| Extract all available slave resources that are identical to the role of the framework group

(This is a slave resource that extracts framework preferences.)

| Determine whether slave resources satisfy the following conditions:

| Filter resources if slave or framework is checkpointing or

Resources are filtered by filters set in the framework. Give up resources.

| Slave is not on the whitelist, give up resources.

| Slave's resources are less than the minimum resource limit and resources are abandoned.

| Add the resources satisfying all the above conditions to the resource result set,

and update slave's available resources (minus resources already placed in the result set)

| if that resource result set is not empty

| Update resources, i.e. available resources in slave

| Execute resource offer operation to deliver resources to framework

The allocator makes a resource offer when:

1. Join the mesos cluster in a new slave.

2. Join the mesos cluster in the new framework.

3. Resource allocation is performed at regular intervals, which can be configured through configuration files.

Allocator provides the following four mechanisms for resource allocation:

Filters: The framework can set filters to reject specific resources. For example, if the framework fails to execute a slave multiple times, the framework can reject the resources on the slave by setting filters on the slave.

White list of slaves: Slaves that are not on the white list do not participate in the resource offer.

Minimum resource constraint of slave: Mesos resource scheduling is equivalent to a binning problem. The wasted space of the boxing problem is related to the ratio of the maximum size of the object to the size of the box. The larger the box, the smaller the object, and the higher the utilization rate. But when a cluster is full of tasks requesting a small amount of resources, then a framework requesting a large amount of resources may starve. In order to accommodate the framework of tasks requiring large resources, mesos avoids offering resources on slaves by setting minimum resource limits on slave nodes until the free resources on slaves reach the minimum resource offering size.

4. Framework preferences for slave resources. Through the role parameter in the resource, dedicated resources and resource preferences for framework groups can be implemented by setting the role value of the slave's resources and grouping the frameworks according to roles.

Some of the shortcomings of Mesos splitter are as follows:

1. Mesos resource offer is essentially a pessimistic concurrency control. From the core scheduling logic, we can see that each scheduling of Mesos will send all available resources to a framework at one time. After receiving the response from the framework, the remaining resources will be returned and the next scheduling will continue. The scheduling performance of Mesos depends on the framework's fast response to resource offers. Mesos does not set a timeout recovery mechanism for resource offers (I read version 0.14 of mesos code and did not find this mechanism). If a framework receives a resource offer and does not respond for a long time, then the entire mesos cluster will deadlock.

Mesos implements fairness by ordering frameworks according to weight and allocated dominant share resources. Mesos If the performance isolation or sharing incentive to achieve dominant resource fairness depends on the implementation of the framework, if there is a greedy framework that consumes a large amount of resources in a resource offer and does not release them for a long time, then the other frameworks will be in a state of starvation. Mesos is suitable for frameworks that use short tasks and have scalable elasticity mechanisms.

Mesos does not support frameworks to preempt resources, and frameworks cannot obtain the state of the entire cluster.

About how to carry out Mesos resource scheduler implementation analysis of the answer to the problem shared here, I hope the above content can be of some help to everyone, if you still have a lot of doubts not solved, you can pay attention to the industry information channel to learn more related knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.