Shulou(Shulou.com)05/31 Report--

This article mainly introduces "what are the main components of ResourceManager". In daily operation, I believe many people have doubts about the main components of ResourceManager. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful for you to answer the questions of "what are the main components of ResourceManager?" Next, please follow the editor to study!

In the hadoop0.23 version, MapReduce has a complete subversion than before, and now we have a YARN called MapReduce 2.0 (MRv2).

The core concept of MRv2 is to separate the two most important functions in JobTrack, resource management and job scheduling/monitoring, to form different daemons. This concept gives rise to a global ResourceManager (RM) and per-application (one for each application) ApplicationMaster (AM). An application is a job in a traditional Map-Reducejobs scenario, or a DAG in jobs (Directed Acyclic Graph directed acyclic graph).

ResourceManager, per-node slave, NodeManager (NM) form data-computation framework. ResourceManager has the highest authority on resource allocation for all application in the system.

Per-application ApplicationMaster is actually a framework specific library, and its task is to schedule resources allocated from ResourceManager and work with NodeManager (s) to execute and monitor tasks.

ResourceManager has two main components: Scheduler and ApplicationsManager.

The Scheduler is responsible for allocating resources to various running applications subject to familiar constraints of capacities, queues etc [J1]. Scheduler is only used for scheduled scenarios, and it does not monitor or track the status of application. Similarly, it cannot restart task after task fails due to application or hardware failure. Scheduler performs its scheduling function based on applications's requirements for resource; it is based on an abstract concept of resource Container, and container is composed of various resources, such as memory, cpu, disk, network, and so on. In the initial version, memory is supported.

Scheduler has a pluggable plug-in that serves to split the cluster resources for various queues, applications, and so on. Current Map-Reduce scheduler such as CapacityScheduler and FairScheduler is such an example.

The CapacityScheduler supports hierarchical queues to allow for more predictable sharing of cluster resources.

ApplicationsManager is responsible for receiving job submissions and assigning the initial container to execute the application,provides the service for restarting the ApplicationMaster container on failure of a specific ApplicationMaster.

NodeManager is a per-machine framework agent that monitors containers's resource (cpu, memory, disk, network) and truthfully report to ResourceManager/Scheduler.

The responsibility of Per-application ApplicationMaster is to combine the resources obtained from Scheduler into the appropriate resource container, track their status, and monitor their process.

MRv2's API is backward compatible with the previous version (Hadoop-0.20.205). This means that all existing Map-Reduce jobs should be able to run on MRv2 without modification just by recompiling it.

At this point, the study on "what are the main components of ResourceManager" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.