Shulou(Shulou.com)06/01 Report--

This article introduces the relevant knowledge of "what are the Hadoop Yarn knowledge points of Resource Management Scheduler?". In the operation of actual cases, many people will encounter such a dilemma. Next, let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Yarn (Yet Another Resource Negotiator) is a resource scheduling platform, which is responsible for allocating resources and scheduling for computing programs such as Spark and MapReduce, and does not participate in the internal work of user programs. It is also the Master/Slave architecture. The global resource manager (ResourceManager) master node, the global resource manager, is responsible for the resource management and allocation of the whole system, mainly composed of the scheduler and the application manager.

The scheduler allocates resources in the system to running applications according to capacity, queues and other constraints, such as how many resources are allocated per queue, a maximum number of jobs, and so on.

The Application Manager (ApplicationsManager) is responsible for managing all applications throughout the system, including the application submission, negotiating resources with the scheduler to start, monitor the Application Master, and restart it and other node resource managers (NodeManager) in case of failure

From the node, the resource and task manager on each node, it needs to report to ResourceManager the resource usage on this node and the running status of each Container, and receives and processes various requests such as Container start / stop from Application Master.

Application Manager (Application Master) each application submitted by the user contains an application manager, and the corresponding MapReduce is MRAppMaster. The main functions include: 1. Request a resource from the global resource manager 2. Further allocate the obtained resources to internal tasks 3. Communicate with Node Explorer to start / stop tasks

4. Monitor the running status of all tasks and re-request resources for the task to restart the task when the task fails

Container

It can be understood that the classes encapsulated in Yarn for the resources that have been applied for by a node, such as memory, CPU, etc., are dynamically generated according to the requirements of the application, and are the basic resource division units in Yarn. Multiple Container exist on one NodeManager node at the same time. Each task that submits the job runs in the Container

According to some restrictions, such as the number of resources allocated per queue and the maximum number of jobs executed, the Yarn scheduler allocates resources in the system to each application according to the resource requirements of the application. The resource allocation unit is the container mentioned above, and users can design a new scheduler according to their own needs. Currently, Yarn also provides a variety of schedulers that can be used directly:

FIFOScheduler (first-in, first-out scheduler): preemption is not supported. If there is a task that runs very slowly, it will affect other tasks FairScheduler (Fair Scheduler): each user shares cluster resources fairly and supports preemption. If a task takes up a large amount of resources for a long time, exceeding the proportion of resources it should allocate, it will terminate the task that gets too many resources CapacityScheduler (capacity Scheduler): a hierarchical queue Each queue allocates a certain amount of capacity (such as assigning small jobs and large job to different queues), and FIFO is supported within a single queue.

The author emphasizes:

1.Yarn does not know the running mechanism of the user submission program, but only provides the scheduling of computing resources (the user program applies for resources from yarn, and yarn is responsible for allocating resources)

The supervisor role in 2.Yarn is ResourceManager, and the role that provides computing resources is NodeManager.

3.Yarn is completely decoupled from the running user programs, which means that various types of distributed computing programs, such as Spark, MapReduce, Storm, Tez, etc., can be run on Yarn, provided that these technical frameworks have resource request mechanisms that conform to Yarn specifications.

4. Because Yarn does not participate in the execution of user programs, Yarn has become a general resource scheduling platform. Various computing engine clusters that previously existed in enterprises can be integrated on a resource management platform to improve resource utilization.

5. The scheduler does not participate in any work related to a specific application, such as monitoring or tracking the execution status of the application, nor is it responsible for restarting failed tasks caused by application execution failure or hardware failure, which are completed by the application-related Application Master.

This is the end of the content of "what are the Hadoop Yarn knowledge points of the Resource Management Scheduler". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.