Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "what is Mesos". In the operation of actual cases, many people will encounter such a dilemma. Then let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Mesos project can easily realize automatic scheduling of distributed applications. At the same time, Mesos itself also highly combines and hosts Docker and other related container technologies. Based on a large number of existing application frameworks of Mesos, user applications can be quickly launched.

Mesos can abstract and schedule the resources of the entire data center (including CPU, memory, storage, network, etc.). Multiple applications run in the cluster at the same time to share resources without paying attention to the distribution of resources in the house.

If you think of the cluster resources in the data center as a server, then what Mesos does is actually the responsibility of today's operating system kernel: abstract resources + scheduling tasks. The Mesos project code is relatively mature. Has many eye-catching features, including:

Large-scale scenarios that support tens of thousands of nodes

Supports a variety of application frameworks, including Marathon, Singularity, Aurora, etc.

Support for HA (based on ZooKeeper implementation)

Support Docker, LXC and other container mechanisms for task isolation

Provides API in several popular languages, including Python, Javaden

Comes with a simple and easy-to-use WebUI, which is convenient for users to operate directly.

Mesos uses the classic "master-slave" architecture, in which the master node (management node) can use Zookeeper to do HA. The Mesos master service award runs on the master node, while the Mesos slave service needs to run on each computing task node. The application framework responsible for completing specific tasks, interacting with Mesos master to apply for resources.

Mesos has three basic components: management service (master), task service (slave), and application framework (framework).

Management Services (master): like most distributed systems, the master node plays a management role, sees global information, and is responsible for resource scheduling and logical control between different application frameworks. The application framework needs to be registered with the management service before it can be used. Users and applications need the API provided by the general master node to obtain the cluster status and operate the cluster resources.

Task service (slave): responsible for reporting the resource status (idle resources, running status, etc.) on the slave node to the master node, and is responsible for isolating local resources to perform specific tasks assigned by the master node. Isolation mechanism currently includes a variety of container mechanisms, including LXC, Docker and so on.

Application Framework (framework): the application framework is actually working, including several major components:

Scheduler: register with the host, waiting for resources to be allocated

Actuators: perform the tasks specified by the framework on the slave node (the framework can also use Mesos's own executors, including shell script executors and Docker executors)

The application framework can be divided into two types: one is that the demand for resources will be expanded (such as Hadoop, Spark, etc.) and may be adjusted after application; the other is that the demand for resources will be fixed (MPI, etc.), one application can be done.

Dispatching

For a resource scheduling framework, the core is the scheduling mechanism. How to allocate the resources of an application framework quickly and efficiently is where the core competitiveness lies. Ideally (which cannot be achieved most of the time), it is best to guess the actual needs of applications and maximize resource utilization.

In order to achieve optimal scheduling, Mesos adopts a two-layer (two-layer) scheduling algorithm.

The basic idea of scheduling is very simple. Master first dispatches a large chunk of resources globally to a framework,framework 's own fine-grained scheduling within the implementation to determine how many resources are used for that task. Two-tier scheduling simplifies the scheduling process of Mesos master itself. By handing over complex fine-grained scheduling to framework, it prevents Mesos master from becoming a performance bottleneck.

Scheduling process

Scheduling interactions through offer delivery. An offer is a set of resources such as >.

The basic scheduling process is as follows:

The Slave node periodically reports its available resources to master.

At some point, master receives a resource request from the application framework and calculates a resource offer to framework according to the scheduling policy.

After receiving the offer, framework can decide whether to want it or not. If it accepts it, return a description of how it wants to use and allocate these resources to run certain tasks (it can be stated that only part of the resources are expected to be used, and the excess will be recovered by master).

Master sends it to slave based on the specific allocation of framework replies to use framework's executor to execute tasks in accordance with the assigned resource policy.

A specific example is given: a slave node reports its free resources to the master node. At the same time, when the master node sees an application framework request, it creates an offer to send the satisfied resources to the application framework. When the application framework (the scheduler) receives the offer and finds it acceptable, it restores the master node and tells the host node that it wants to run two tasks: one occupancy and one occupancy. After receiving the task information, the master node assigns the task to run on the slave node (in fact, the executor of the application framework is responsible for executing the task). Resources can be released at the end of the task. The rest of the resources continue to be allocated to other application frameworks or tasks.

After receiving the offer, if the offer does not meet its preferences (such as wanting to continue to use the last slave), the application framework can choose to reject the offer and wait for the master to send a new offer. In addition, the process of resource allocation can be accelerated through the filtering mechanism.

This is the end of "what is Mesos". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.