Shulou(Shulou.com)05/31 Report--

This article mainly explains "what is the working mechanism of YARN cluster". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "what is the working mechanism of YARN cluster".

Description: YARN is the addition of Hadoop2.0 to a resource scheduling framework, which is also the master-slave node architecture: ResouceManager (Resource Management) and NodeManager.

Advantages:

1. Separate resource scheduling and task monitoring to solve the bottleneck of JobTracker (stress, single point of failure)

2. General resource scheduling platform, compatible with more computing frameworks (storm/spark/mapreduce)

Working mechanism:

(the picture is taken from the official website. The picture on the official website looks very complicated, which is very difficult for me to look at.)

How Yarn clusters work:

Before understanding the working mechanism of Yarn, we should know that ResourceManager (hereinafter referred to as RM) is a resource task scheduler, which is responsible for the allocation of task resources.

1. Before the task is executed:

1.1.The job job is submitted to the Yarn cluster first to RM.

1.2. RM returns the resource path related to job_id and job

1.3. upload job to HDFS after obtaining the resource path

1.4.Notification RM,job to finish uploading

RM will initialize the job task after receiving the job and submitting it.

1.6. add job tasks to the task list (task list)

NodeManager (hereinafter referred to as NN) periodically maintains heartbeat communication with RM, and then checks to see if there are any tasks that need to be picked up.

1.8.Once NN checks a task that needs to be picked up by itself, it will pick up the task

1.9. After receiving the task, a resource container of a certain size will be allocated according to the job resource information.

1.10. Download job resources to the container

2. Perform tasks:

Yarn provides AppMaster interface, AppMaster mainly monitors and schedules tasks, RM only manages AppMaster, and once AppMaster is down, it will be restarted on other NN nodes, so the pressure on RM is reduced (compared with Hadoop1.0 's JobTracker). The computing framework can run on Yarn cluster as long as it implements this interface, which is also the second benefit of Yarn cluster, which is compatible with more computing frameworks such as Spark/Storm/MapReduce. The interface implemented by MapReduce is MRAppMaster

2.1. start MRAppMaster (RM will start on a NN node according to the situation)

2.2. MRAppMaster registers with RM

2.3.The MRAppMaster starts the Map Task of each NN node according to the information provided by RM

2.4.Starting Reduce Task of each NN node

2.5. initiate a logout operation to RM after the task has been executed to destroy yourself

Thank you for your reading, the above is the content of "what is the working mechanism of YARN cluster". After the study of this article, I believe you have a deeper understanding of what the working mechanism of YARN cluster is, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.