Shulou(Shulou.com)05/31 Report--

How to understand the working mechanism of Yarn, in view of this problem, this article introduces the corresponding analysis and answer in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

We know that YARN is a Hadoop explorer, and Yarn should be used in both MapReduce and Spark applications.

Today we're going to talk about the composition and workflow of Yarn:

Introduce a few roles first.

ResourceManager

A cluster has only one RM, which is the commander-in-chief of YARN and is responsible for coordinating computing resources on the cluster. It has the following two components:

ApplicationsManager: this is not ApplicationMaster. Pay attention to the distinction. It accepts requests from Job, dispatches Container to start ApplicationMaster, and is responsible for failed restarts of ApplicationMaster.

Scheduler: the scheduler simply allocates resources to the application. It does not monitor the status of the application. Scheduling here is based on Container, an abstract resource container, including memory, CPU, disk, network, and so on.

NodeManager

There are multiple NodeManager in a cluster, which is responsible for launching Container, monitoring Container resource usage (cpu, memory, disk, network), and reporting this information to RM. There can be more than one Container on a NodeManager.

Container

Container is an abstraction of a set of hardware resources, including CPU, memory, disk, network, etc. All Job runs in Container

ApplicationMaster

Negotiate resources with RM and monitor tasks with NodeManager, which, like ApplicationMaster and Job, runs in Container.

Let's take a closer look at how the next task is submitted to Yarn to run.

Step 1: Client initiates a task request to RM: "RM, I want to perform a task"

Step 2: RM returns information such as Application ID to Client: "all right, I'll assign you an Application ID."

Step 3: according to the returned information, Client checks whether the input and output directory of the job is correct, calculates whether the job input shard is normal, and copies the resources (job Jar file, configuration file, input shard) needed to run the job to the HDFS directory named Application ID.

Step 4: Client formally submits the assignment to RM: "I'm ready to carry out the task."

Step 5: RM sends the request message to his younger brother Scheduler: "Brother Scheduler, you assign a Container."

Step 6: Scheduler assigns Container to start ApplicationMaster: "OK, this Container is contestant 0"

Step 7: ApplicationsManager communicates with the specified NodeManager, requiring ApplicationMaster to be started in Container.

Step 8: ApplicationMaster initializes the task and requests the required resources from RM: "Brother RM, give me the resources to run the task"

Step 9: RM returns the resource requested by ApplicationMaster: "give you one or two NodeManager"

Step 10: ApplicationMaster communicates with the corresponding NodeManager and applies for the Container startup task "Brother NodeManager, start Container for me to run this task"

Step 11: applications in Container will first download the required computing resources from HDFS to local, and then start the task: "run for me."

Step 12: during the run, the task reports status and progress to ApplicationMaster,Client and polls ApplicationMaster for status.

Step 13: after the run is complete, Container will log out, that is, the resources will be returned to the system, and ApplicationMaster will log itself out to RM.

This is the answer to the question on how to understand the working mechanism of Yarn. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel to learn more about it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.