Shulou(Shulou.com)06/02 Report--

This article mainly introduces "Apache Mesos overall architecture analysis". In daily operation, I believe many people have doubts about Apache Mesos overall architecture analysis. The editor consulted all kinds of data and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the doubts about "Apache Mesos overall architecture analysis". Next, please follow the editor to study!

1. Preface

Like most other distributed systems, Apache Mesos also adopts master/slave structure in order to simplify the design. In order to solve the master single point fault, master is made as lightweight as possible, and all the above meta-numbers can be reconstructed through each slave re-registration, so it is easy to solve the single point failure problem through zookeeper.

2. Interpretation of basic terms in Apache mesos

(1) Mesos-master:Mesos master, which is mainly responsible for managing each framework and slave and allocating resources on the slave to each framework

(2) Mesos-slave:Mesos slave, which is responsible for managing each mesos-task on this node, such as allocating resources to each executor.

(3) Framework: computing framework, such as Hadoop,Spark, which is connected to Mesos through MesosSchedulerDiver

(4) Executor: executor, installed on mesos-slave, used to start task in the computing framework.

When users try to add a new computing framework to Mesos, they need to implement a Framework scheduler and executor to access Mesos.

3. Overall architecture

Apache Mesos consists of four components, Mesos-master,mesos-slave,framework and executor.

Mesos-master is the core of the whole system, which is responsible for managing each framework (managed by frameworks_manager) and slave (managed by slaves_manager) connected to mesos, and allocates resources on slave to framework (managed by independent plug-in module Allocator) according to a certain policy.

Mesos-slave is responsible for receiving and executing commands from mesos-master, managing mesos-task on nodes, and allocating resources to each task. Mesos-slave sends its own amount of resources to mesos-master, and the Allocator module in mesos-master decides which framework to allocate resources to. There are two resources under consideration: CPU and memory, that is, mesos-slave will send the number of CPU and the amount of memory to mesos-master, and when users submit jobs, they need to specify the number of CPU and the amount of memory required for each task, so that when the task runs, Mesos-slave runs tasks in a linux container that contains fixed resources to achieve the effect of resource isolation. Obviously, there is a single point of failure problem in master, so mesos uses zookeeper to solve this problem.

Framework refers to external computing frameworks, such as Hadoop,Mesos, which can be connected to mesos through registration, so that mesos can carry out unified management and resource allocation. Mesos requires that the accessible framework must have a scheduler module, which is responsible for task scheduling within the framework. When a framework wants to connect to mesos, it needs to modify its own scheduler to register with mesos and obtain the resources assigned to it by mesos, so that its own scheduler allocates these resources to tasks in the framework. In other words, the entire mesos system adopts a two-tier scheduling framework: * * layer, where mesos allocates resources to the framework; second, the framework's own scheduler allocates resources to its own internal tasks. At present, Mesos supports schedulers written in three languages, namely, Cellular scheduler java and python. In order to provide unified access to various schedulers, Mesos uses C++ to implement a MesosSchedulerDriver (scheduler driver). Framework's scheduler can call the interface in the driver to interact with Mesos-master to complete a series of functions (such as registration, resource allocation, etc.).

Executor is mainly used to launch task within the framework. Because different frameworks have different interfaces or ways to start task, when a new framework wants to connect to mesos, you need to write an executor to tell mesos how to start task in the framework. In order to provide a unified way to write actuators for various frameworks, Mesos uses C++ to implement a MesosExecutorDiver (Actuator driver). Framework can tell mesos how to start task through the relevant interfaces of this driver.

At this point, the study on the "overall architecture analysis of Apache Mesos" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.