Shulou(Shulou.com)06/02 Report--

This article mainly explains "what is the working principle of big data Apache Hadoop YARN". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "what is the working principle of big data Apache Hadoop YARN"?

I. the basic composition of Apache Hadoop YARN

Master/Slave structure: 1 ResourceManager (RM) corresponds to multiple NodeManager (NM); YARN is composed of Client, ResourceManager, NodeManager, ApplicationMaster (AM); Client submits tasks to RM, kills tasks, etc.; AM is completed by corresponding applications; each application corresponds to an AM,AM to RM for resources to launch the corresponding Task;NM on NM to RM through heartbeat information: report NM health status, task execution status, pick up tasks, etc.

The resource management module YARN of the Hadoop of big data series

1. RM: there is only one cluster, which is responsible for the unified management and scheduling of cluster resources:

Process requests from the client (start / kill the application); start / monitor AM; once an AM dies, the RM will start the AM; monitoring NM on another node, receive the heartbeat report from the NM and assign tasks to the NM to execute; once a NM dies, mark the tasks on the NM to tell the corresponding AM how to deal with it; responsible for resource allocation and scheduling of the entire cluster

2. NM: there are multiple in the whole cluster, which are responsible for the management and use of single-node resources.

Periodically report to RM the resource usage on this node and the operation of each Container; receive and process various commands from RM Container start / stop; process commands from AM; responsible for resource management and task scheduling on a single node

3. AM: one for each application, responsible for the management of the application

Data segmentation; request resources (Container) from RM for applications / jobs and assign them to internal tasks; communicate with NM to start / stop tasks; task monitoring and fault tolerance (re-request resources for the task to restart tasks when task execution fails); process commands sent by RM: kill Container, restart NM, etc.

4. Container: abstraction of the running environment of a task

Task running resources (nodes, memory, CPU); task startup commands; task running environment; tasks are running in Container, and a Container can run either AM or specific Map/Reduce/MPI/SparkTask

Second, the working principle of YARN

Users submit applications / jobs to YARN, including ApplicaitonMaster programs, commands to start ApplicationMaster, user programs, etc. ResourceManager assigns the first Container to the job and communicates with the corresponding NodeManager, requiring it to start the ApplicationMaster;ApplicationMaster of the job in this Containter to register with ResourceManager first. In this way, users can query the running status of the job through ResourceManager. It will then request resources for each task and monitor the running status of the task until the end of the run. That is, repeat steps 4-7. The ResourceManager ApplicationMaster uses polling to apply for and receive resources from ResourceManager through RPC requests.

Once the ApplicationMaster has applied for resources, it communicates with the corresponding NodeManager, asking it to start the task; the NodeManager starts the task; each task reports its status and progress to the ApplicationMaster through the RPC protocol, so that the ApplicaitonMaster can keep abreast of the running status of each task at any time, so that the task can be restarted when the task fails; during the running of the job, the user can query the current running status of the job from ApplicationMaster through RPC at any time When the job is complete, ApplicationMaster logs out to ResourceManager and closes himself

III. Fault tolerance of YARN

ResourceMananger implements HA to avoid a single point of failure based on ZooKeeper; after NodeManager execution fails, ResourceManager tells the corresponding ApplicationMaster; that the failed task will be decided by ApplicationMaster; after ApplicationMaster execution fails, ResourceManager will restart it; ApplicationMaster needs to deal with the fault tolerance of internal tasks; RMAppMaster will save the Task that has already been run, and there is no need to run it again after restart.

Fourth, YARN scheduling framework

1. Two-tier scheduling framework

ResourceManager allocates resources to ApplicationMaster;ApplicationMaster and further allocates resources to individual TASK

2. Scheduling strategy based on resource reservation.

When resources are insufficient, Task will be reserved until resources are sufficient. Description: when a Task needs 10G resources and each node is less than 10G, then select a node, but only 2G on a NodeManager, then reserve resources on this NodeManager. When other resources are released on this NodeManager, resources will be reserved for 10G jobs until 10G is saved and Task is started. Disadvantages: the resource utilization rate is not high, so you have to save it first and wait until 10G to make use of it, resulting in low resource utilization of the cluster.

Different from the description of the "all or nothing" strategy: when a job requires 10G resources, the nodes wait less than 10G and wait until there are 10G free resources on a node, which is likely to cause the Task to shut down.

Thank you for your reading, the above is the content of "what is the working principle of big data Apache Hadoop YARN". After the study of this article, I believe you have a deeper understanding of what the working principle of big data Apache Hadoop YARN is, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.