Shulou(Shulou.com)06/03 Report--

Yarn is a second-generation MR running architecture designed to solve the heavy load of JobTracker in MRv1.

Yarn is a master/slave structure, and the master node runs the Resourcemanager,slave node and runs nodemanager.

RM has two main components: resource Scheduler (scheduler) and Application Manager (Applications Manager)

Scheduler mainly dispatches resources and does not take any action related to the application, including Capacity Scheduler and Fair scheduler.

ASM: mainly responsible for submitting, launching, and monitoring the application masterAM of each application

Each job has an application master to monitor and manage all its own tasks, with the following features:

A, interact with RM to get the resources for the task to run

B, interactive with NM to submit tasks, monitor the operation of tasks and other NM

Generally, each node in the cluster starts a NM,NM not only to send a heartbeat to RM and report the resource usage of this node, but also to interact with AM to receive and process all requests for container from AM. Container

A logical concept, an abstract representation of resources, can represent a set of multi-dimensional resources. Currently, yarn mainly supports cpu and memory. Yarn workflow

After submitting an application to yarn, yarn is divided into two steps: the first step is to start application master;, and the second step is to start and monitor all tasks in this application by AM until completed.

Submit the application to yarn, including the application master program, and the command to start am; RM assigns a container to AM and communicates with NM to start container to run the AM (this is the only container initiated by RM and NM communication, other container is started by AM and NM communication) after AM starts, it actively sends a heartbeat to RM to register itself, at this time RM can monitor the AM, and we can see this task on the yarn web page AM polling applies for and obtains resources from RM; after AM gets the resources, it communicates with NM to start the task; NM sets the running environment of the task to start the task; tasks on each NM constantly send heartbeats to AM through RPC to report their status and progress; after all the tasks are completed, AM logs itself off to RM.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.