Shulou(Shulou.com)06/03 Report--

Monday, 2019-2-18

Summary of yarn knowledge system

The cause of Yarn

(1) in MapreduceV1, jobtracker has a bottleneck:

All mr programs running on the cluster are scheduled by jobtracker

SPOF single point of failure

The division of responsibilities is not clear

(2) divide the responsibilities of jobtracker into two parts:

 resource scheduling and management: implemented by a unified resource scheduling platform (cluster) (yarn)

 task monitoring and management:

Each application runtime has its own task monitoring and management process AppMaster

B. Life cycle of AppMaster: after the application is submitted to the yarn cluster, yarn is responsible for starting the AppMaster of the application, and then the task execution monitoring and scheduling are handed over to AppMaster. When the application is finished, AppMaster logs itself out to yarn.

The specific implementation of C, AppMaster is responsible for the distributed computing framework used by application. For example, application of Mapreduce type has MrAppMaster implementation class. Spark DAG applications have SparkContext implementation of SparkOnYarn.

Architecture diagram of yarn

2 main class nodes (service processes) of yarn

1. Resourcemanager master node master-only 1 is needed (HA can also be implemented)

2. Nodemanager slave nodes-there can be multiple slave nodes depending on the size of the cluster

The specific summary is as follows:

(1) ResourceManager-> master node, which can be configured with multiple RM to implement HA mechanism.

ResourceManager consists of two core components:

Scheduler and ApplicationsManager

Scheduler: responsible for resource scheduling, scheduling policy pluggable (built-in implementation CapacityScheduler / FairScheduler) does not provide monitoring of the operation of application

ApplicationsManager: responsible for responding to task submission requests, negotiating the container that the applicationMaster is running, and restarting the failed applicationMaster

(2) NodeManager-> slave nodes, one on each machine

Responsibilities: load containers, monitor resource usage of each container, and report to Resourcemanager/Scheduler

(3) ApplicationMaster-> specific computing framework is implemented by itself, and the interface is unified AppMaster.

Responsibilities: request appropriate resources from Scheduler, track task execution, monitor task execution progress, status, etc.

The basic concept of yarn-the process of mr program submission

Yarn:hadoop Resource Management and scheduling platform (Cluster)-provide computing resource management and scheduling for user programs

User programs: such as mr programs developed by users

Detailed process of mr program submission

1. After runjar starts, it will find resourcemanager and issue a remote procedure call RPC / / to submit a job (mr program)

2. Resourcemanager will return a jobid and a hdfs path to submit job-related files to runjar programs such as hdfs://../stagingDIR

3. Runjar will upload job-related files to the hdfs path, such as hdfs://.../stagingDIR/jobid/worcount.jar.

Jbo.xml

Slice information of job split

4. Notify RM that the file is uploaded successfully

5. RM will construct a task for the task submitted by the client and put it in his internal task queue. Tasks submitted by other clients will also be in this queue and will wait when resources are insufficient.

6. There is a heartbeat communication between nodemanager and resourcemanager at all times. After the task comes out, nodemanager will automatically pick up the task. For example, when nodemanager receives the task, one of the nodes will get the task, and nodemanager will pull the job-related files on hdfs.

7. Nodemanager isolates part of the resources inside itself to form a container cpu+mem. The first thing to do in this task is: he will start a program in the container. (then how does he know to start this program? Resource manager will be described in this job task will embed a startup program shell command) nodemanager is responsible for the shell command on his machine to execute on the ok, execute the following will start a MapReduce jar package in the container of a program, this program is called (MRappmaster process) and then all the processing flow will be dominated by this MRappmaster process.

What would MRAPPmaster do? What is he himself?

MRappmaster: actually, he is the director of an MapReduce application. The supervisor process is implemented by the MapReduce framework, and this is used to manage the execution of the MapReduce program.

In yarn, it is only responsible for starting the mrappmaster process. Yarn doesn't care exactly how other MapReduce programs run. Yarn is only responsible for giving you the resources you need, so Ok.

8. After mrappmaster starts, it registers with resourcemanager and applies for resources to run the program. Mrappmaster starts the map task and reduce task processes, so resources are required, including those needed to monitor and manage tasks.

9. Resourcemanager will assign resources to the response requested by mrappmaster, and new tasks will be created in the task queue. These tasks will be handled by nodemanager (that is, tasks for allocating resources). When nodemanager receives a new task for allocating resources, nodemanager will create a new container on its own machine (multiple containers can be created on one machine). The list of container resources created by each nodemanager will be returned to RM

10. RM will return the resource list information to MRappmaster

11. Mrappmaster will find the machine in each nodemanager that gets the task and create the container, and ask them to start the map task process. (to start several map task processes, you can see the job split slicing rules in the job task. There are several map task processes in the rules.)

12. When the map task processing completes the task, it will return a message to mrappmaster to tell him where I put the results of the processed task.

13. Mrappmaster will apply to RM for new resources again, and RM will assign a task to the task queue for this application, then one of the nodemanager will pick up the task and assign the container, then it will return to RM to tell him that I have created the container, and then RM will tell MRappamster that the resources you need, I will create it for you, and tell him which nodemanager is on it.

14. Mrappmaster starts a reduce task process in the nodemanager container of the response and tells him where the map output is.

Reduce task will take it and carry out his internal execution process.

16. Reduce task will return the processing result to mrappmaster

17. The whole MR program is processed, and mrappmaster logs itself out to RM.

Summary:

The working Mechanism of YARN-- the running process of MR Program

Check the process of map task and reduce task. What you see with the command jps is yarn child.

Summary:

1.yarn only responds to the submission of job and allocates resources for the operation of job

Yarn does not participate in the specific operating mechanism and process of job

There is a MrAppmaster in the 2.mapreduce program to control the specific running process of the program.

3. Since yarn does not provide the specific operation of user programs, yarn can provide resource services for various types of resource programs.

(YARN is similar to an operating system platform that can run programs developed by various computing frameworks: mapreduce storm spark...)

4. Because of this operating mechanism of yarn, the life cycle of hadoop is prolonged.

5. Because yarn can integrate a variety of computing frameworks, enterprises can integrate various distributed frameworks into hadoop clusters

In order to facilitate the enterprise management and save the cost of the enterprise.

Application lifecycle

Yarn supports short-cycle and long-cycle applications

MR: a short-cycle application in which each job of the user is an application

Spark: short-cycle application, but more efficient than the previous one. It converts a workflow (DAG) into an application, so that container can be reused between job and intermediate result data can be reused without landing.

Storm:long-running application, which is shared by multiple users, reduces the early consumption of resource scheduling, thus providing users with low latency response.

Resource request

The resource request is described by the Container object and supports data locality constraints. For example, if the data on the hdfs is processed, the container is first allocated to the datanode where the block is located. If the datanode resource does not meet the requirements, it is preferred to be in the same rack, and if it cannot be met, it is allocated randomly.

Application can request resources at any stage of its life cycle, can request all the resources needed at the beginning, or dynamically request resources during operation. For example, spark adopts the first strategy, while MR is divided into two phases. The resources of map task are requested once at the beginning, while the resources of reduce task are dynamically requested during the running process. Moreover, after the task fails, you can also rerequest resources for retry.

Yarn task scheduling-capacity scheduler (capacity scheduling) / fair scheduler (fair scheduling) / / see separate documentation for this section

Due to the limited cluster resources, when many application resource requests cannot be satisfied, yarn needs appropriate policies to schedule application resource requests.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.