Shulou(Shulou.com)06/03 Report--

Yarn is a distributed resource management system to improve the resource utilization in the distributed cluster environment, including memory, IO, network, disk and so on. The reason is to solve the deficiency of the original MapReduce framework.

At first, the committer of MapReduce can periodically modify the existing code, but with the increase of the code and the deficiency of the original MapReduce framework design, it becomes more and more difficult to modify the original MapReduce framework.

So the committer of MapReduce decided to redesign MapReduce from the architecture, so that the next generation MapReduce (MRv2/Yarn) framework has better scalability, availability, reliability, backward compatibility and higher resource utilization, and can support more computing frameworks in addition to the MapReduce computing framework.

1. The deficiency of MapReduce framework.

Now it is more popular to say that there is a problem with jobtracker, such as a single point of failure and heavy tasks. But in addition to Jobtracker, there is also a TaskTracker. Let's take a look at the following picture:

Overview of JobTacker

The tasks undertaken by JobTacker are: accepting tasks, computing resources, allocating resources, and communicating with DataNode.

In hadoop, each application is represented as a job, and each job is divided into multiple tasks. The job control module of JobTracker is responsible for job decomposition and status monitoring.

* the most important thing is status monitoring, including TaskTracker status monitoring, job status monitoring and task status monitoring. Main role: fault tolerance and provide decision-making basis for task scheduling.

Overview of TaskTracker

TaskTracker is the bridge between JobTracker and Task: on the one hand, it receives and executes various commands from JobTracker: running tasks, submitting tasks, killing tasks, etc.; on the other hand, it periodically reports the status of each task on the local node to JobTracker through the heartbeat. TaskTracker communicates with JobTracker and Task using RPC protocol.

Resource slot Overview

Slot is not the Core of CPU, nor is it memory chip, it is a logical concept. The number of slot of a node is used to represent the capacity or capacity of a node's resources, so slot is the resource unit of Hadoop.

What is slots in hadoop

Therefore, there are too many tasks that JobTracker needs to complete, not only to maintain the state of job but also to maintain the state of task of job, resulting in excessive resource consumption.

On the taskTracker side, using map/reduce task as the representation of resources is too simple to take into account CPU, memory and other resources. When two task that consume a lot of memory are scheduled together, OOM is easy to appear.

Resources are forcibly divided into map/reduce slot. When there is only map task, reduce slot cannot be used; when there is only reduce task, map slot cannot be used, which can easily lead to underutilization of resources.

2 、 Yarn

First, let's take a look at the architecture of Yarn

Overview of 1.ResourceManager

Is global, responsible for having the highest control over all resources in the system. ResourceManager, as the resource coordinator, has two main components: Scheduler and ApplicationsManager (AsM).

Scheduler is responsible for allocating the minimum amount of resources needed to run application to Application. Scheduler is only scheduled based on resource usage and is not responsible for monitoring / tracking the status of application and certainly does not deal with failed task.

ApplicationsManager is responsible for processing the job submitted by client and negotiating the first container for applicationMaster to run, and restarts applicationMaster if applicationMaster fails.

Overview of 2.NodeManager

NM is mainly responsible for starting container assigned by RM to AM and container representing AM, and monitors the operation of container.

When starting container, NM will set some necessary environment variables and download the jar packages and files needed for container to run locally from hdfs, that is, the so-called resource localization; when all the preparatory work is done, it will start the script on behalf of the container to start the program.

After startup, NM will periodically monitor the resources consumed by the container. If it exceeds the amount of resources declared by the container, it will kill the process represented by the container.

Overview of 3.ApplicationMaster

Because NodeManager needs resources to perform and monitor tasks, it communicates with ResourceManager through ApplicationMaster to obtain resources. In other words, ApplicationMaster acts as a middleman.

To put it in more technical terms: AM is responsible for asking ResourceManager for the resource containers that NodeManager needs to perform tasks, and more specifically, ApplicationMaster is responsible for requesting resources from Scheduler and tracking the use of these resources and monitoring the progress of tasks.

So we see that the functions of JobTracker are distributed among various processes, including ResourceManager and NodeManager:

For example, the monitoring function is distributed to NodeManager, and Application Master.

ResourceManager is divided into two components: the scheduler and the application manager.

That is to say, after Yarn refactoring, the functions of JobTracker are distributed to various processes. At the same time, because these processes can be deployed separately, this greatly reduces the pressure of a single point of failure.

Finally, to remind you that writing applications on yarn is different from familiar MapReduce applications, we must keep in mind that yarn is only a resource management framework, not a computing framework, and computing frameworks can run on yarn. All we can do is apply to RM for container, and then work with NM to launch container.

Conclusion

Thank you for watching. If there are any deficiencies, you are welcome to criticize and correct them.

In order to help you make learning easier and efficient, we will share a large number of materials free of charge to help you overcome difficulties on your way to becoming big data engineers and even architects. Here to recommend a big data learning exchange circle: 658558542 welcome everyone to enter × × × stream discussion, learning exchange, common progress.

When you really start learning, it is inevitable that you do not know where to start, resulting in inefficiency that affects your confidence in continuing learning.

But the most important thing is not to know which skills need to be mastered, step on the pit frequently while learning, and eventually waste a lot of time, so it is necessary to have effective resources.

Finally, I wish all the big data programmers who encounter bottle disease and do not know what to do, and wish you all every success in the future work and interview.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.