Shulou(Shulou.com)06/01 Report--

This article will explain in detail the example analysis of the architecture and implementation of the distributed cluster scheduling framework Mesos. The content of the article is of high quality, so the editor shares it for you as a reference. I hope you will have some understanding of the relevant knowledge after reading this article.

Brief introduction

Mesos is a management system that shares cluster resources in multiple cluster computing frameworks. It improves the utilization of cluster resources and avoids data replication in each computing framework.

Fine-grained resource allocation is achieved through a distributed two-tier scheduling model: Mesos decides how many resources are provided for each framework, the framework decides which resources to accept, and where to assign computing tasks to execute.

Problems and solutions

In the 2010s, the computing framework blossomed, and MapReduce [1], MPI, Dryad, Pregel and so on appeared one after another. Obviously, new cluster computing frameworks will continue to emerge, and there is no cluster computing framework that meets the needs of all applications, so enterprises and research institutions will run multiple computing frameworks in the same cluster. Because different computing frameworks are independent of each other, it is extremely difficult to share data and computing resources among different frameworks.

Mesos implements a lightweight resource sharing layer to ensure fine-grained resource sharing among different frameworks. It should not only meet the needs of different computing frameworks, but also meet the needs of some new computing frameworks in the future. Therefore, there are high requirements for the expansibility and efficiency of Mesos.

One option is a centralized scheduler, which is too complex to meet the needs of all computing frameworks, and will lead to a lot of refactoring, which is not realistic.

Mesos proposes a new scheduling abstraction resource offer,Mesos to determine how many resources are provided to the framework, and the framework decides which resources to accept and match tasks with resources. This is a decentralized scheduling model that is simple and easy to implement, brings high scalability and robustness to Mesos, and has two additional advantages:

1. Support for running different versions of the same framework. Easy to develop new framework

Philosophy of architecture design

The original goal is to provide scalable and resilient kernels for different frameworks, allowing them to share clusters efficiently. However, considering the diversity and fast iteration of the computing framework, the design philosophy evolved into a minimized interface to ensure efficient resource sharing between frameworks, so task scheduling and execution are left to the framework.

Through the resource offer model, Mesos master provides the available resources in the cluster to the framework, and how many resources are provided is based on different policies, such as fairness policy, priority policy and so on. Plug-ins are also provided to allow the framework to implement its own policies.

Each computing framework consists of two parts: scheduler (scheduler) and executor (executor). The scheduler registers with the Mesos master to receive resources, and the executor runs on the salve node to perform specific tasks. The framework does not specify how many resources it needs, but reports Mesos master, uses it when appropriate, and rejects it if it is not appropriate.

The problem with the above mechanism is that if a framework needs a lot of resources and is not satisfied for a long time, hunger will occur (the probability is very low). For this reason, Mesos provides a filters mechanism for the framework, and the framework tells Mesos that it explicitly rejects certain resources.

Fault tolerance

Mesos master retains only a small amount of necessary state information so that when master crashes, the new master can be quickly recovered based on the information on the slave and the framework. Master contains only three types of information: slave, framework, and running tasks.

When running multiple master, use zookeeper to implement leader election.

Code implementation

Mesos is based on C++ and borrows many existing technical achievements, such as C++ actor programming model library libprocess [2], ZooKeeper, Linux Container [3] and so on.

In order to prove that Mesos is lightweight and easy to develop the framework, the author developed a framework Spark for iterative jobs in machine learning based on Mesos, which is the big data processing framework that became famous in the industry. Thanks to the good design of Mesos, the Spark prototype uses only about 1300 lines of code.

Summary

The emergence of Mesos has its specific background and purpose, and it solves the problem of pain points in a certain area of the industry. Mesos implements a distributed two-level scheduling model, which enables different computing frameworks to use resources in the same computer cluster and improves the utilization of cluster resources.

In addition, this framework also led to the birth of another famous framework, Spark.

Mesos comes from UC Berkeley AMPLab. From this paper, we can see the remarkable characteristics of UC Berkeley computer research: the combination of theory and practice, not on paper. This requires researchers (tutor, doctor, master) not only to have first-class research ability, but also to have strong programming practice ability.

So much for the example analysis of the Mesos architecture and implementation of the distributed cluster scheduling framework. I hope the above content can be helpful to you and learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.