Shulou(Shulou.com)06/02 Report--

How to understand the Framework and Executor registration process of Mesos, aiming at this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

Taking Hadoop framework as an example, this paper introduces the registration process of framework and executor to mesos.

1. Framework registration process

(1) when JobTracker starts, the start () method of MesosScheduler is called.

(2) the start () method of MesosScheduler creates a MesosSchedulerDriver object and passes itself as a parameter to the object.

(3) initialize MesosSchedulerDriver and create a SchedulerProcess object

(4) initialize MesosSchedulerDriver and call MasterDetector::create (), which will send a NewMasterDetectedMessage message to the SchedulerProcess object.

(5) after receiving the NewMasterDetectedMessage message, the SchedulerProcess object sends a RegisterFrameworkMessage message to Master

(6) after receiving the message, Master saves the relevant information and returns the FrameworkRegistedMessage message to confirm the success of framework registration.

2. Executor registration process

This section describes the process by which the framework frameworkX registers the executor executorX on a slaveX:

(1) Master*** sends the message RunTaskMessage executing task in frameworkX to slaveX for the second time.

(2) after slave receives the message, run the corresponding message handling function runTask ()

(3) if the function finds that the executorX corresponding to frameworkX is not started on the slave, it calls the lauchExecutor () function of IsolationModule.

(4) this function creates a FrameworkExecutor object, calls the Initialize () function of ExecutorProcess to initialize, and starts TaskTracker at the same time

(5) the Initialize () function creates the message RegisterExecutorMessage and sends it to slave.

(6) after receiving the message, Slave calls the object's message handling function registerExecutor, which creates an ExecutorRegisteredMessage message and returns it to ExecutorProcess.

(7) after receiving the message, ExecutorProcess calls the corresponding message handling function registered (), which further calls the registered () function of FrameworkExecutor.

Next, the RunTaskMessage messages sent by master to slave go through the process shown in the following figure. It should be noted that for the same computing frame, Mesos will only create one resource container on a slave, and all task will run in this container, that is, mesos cannot achieve task-level isolation, only executor-level isolation, while for the same framework, all task on the same slave will all run in the same executor.

For Hadoop, only one TaskTracker is created on each mesos-slave, and the TaskTracker is placed in an executor (corresponding to a linux container) to run, while all task on the same TaskTracker are in the process tree where the TaskTracker resides, so the resources corresponding to the executor are shared. When a new task is received by the TaskTracker, the amount of resources available to the executor is increased (using "lxc-cgroup-n% s% s% lld"), and when the task is finished, the amount of resources available to the executor is reduced (using "lxc-cgroup-n% s% s% lld").

This is the answer to the question about how to understand the Framework and Executor registration process of Mesos. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel to learn more about it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.