Shulou(Shulou.com)06/03 Report--

[TOC]

Yarn process

You can understand it by referring to the illustration and introduction in the authoritative Guide to Hadoop P79. Of course, it will be clearer and easier to understand by combining the following illustration with the explanation in it. Of course, the basic reference individual thinks that the illustration in the book should be more concise. The following can be said to be a detailed version:

1) users submit applications to YARN, including ApplicationMaster programs, commands to start ApplicationMaster, user programs, and so on.

2) ResourceManager assigns the first Container to the application and communicates with the corresponding NodeManager, asking it to start the application's ApplicationMaster in this Container.

3) ApplicationMaster first registers with ResourceManager, so that users can view the running status of the application directly through ResourceManage, and then it will apply for resources for each task and monitor its running status until the end of the run, that is, repeat step 4x7.

4) ApplicationMaster applies for and receives resources from ResourceManager through RPC protocol by polling.

5) once the ApplicationMaster requests a resource, it communicates with the corresponding NodeManager, asking it to start the task.

6) after NodeManager sets up the running environment for the task (including environment variables, JAR packages, binary programs, etc.), write the task startup command into a script, and start the task by running the script.

7) each task reports its status and progress to ApplicationMaster through a certain RPC protocol, so that ApplicationMaster can keep abreast of the running status of each task, so that it can restart the task when it fails. During the running of the application, users can query the current running status of the application to ApplicationMaster at any time through RPC.

8) when the application is finished, ApplicationMaster logs out to ResourceManager and closes itself.

Yarn compared with MapReduce 1 MapReduce 1YarnJobtrackerResourceManager, Application Master, TimeLine ServerTrasktrackerNodeManager, SlotContainer

TimeLine Server, the timeline server, is used to record historical jobs.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.