Shulou(Shulou.com)06/01 Report--

This article introduces the relevant knowledge of "what are the relevant knowledge points of YARN". In the operation of actual cases, many people will encounter such a dilemma. Next, let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Background of YARN production

Why is YRAN generated? This is related to the architecture of MapReduce1.x, and it is precisely because there are many problems in MapReduce1.x that YARN is generated.

The architecture of MapReduce1.x is as follows:

Architecture of MapReduce1.x

When Hadoop1.x, the architecture of MapReduce is still the master-slave architecture. A JobTracker has multiple TaskTracker, the master node is JobTracker, only one slave node is TaskTracker, and there can be multiple slave nodes. The slave node tells it its own operation by sending heartbeat information (heartbeat) to the master node, while the master node is responsible for managing the scheduling.

JobTracker (JT): responsible for resource management and job scheduling.

TaskTracker (TT): regularly report the health status, resource usage and job execution of this node to JT. Receive commands from JT to start the task or kill the task.

So what's wrong with this architecture?

First of all, there is the problem of single point of failure, all the slave nodes (TT) are directly related to the master node (JT). If the master node accidentally dies, then the whole system will collapse and there will be no way to run.

Secondly, JT is under great pressure and is not easy to expand. He has to receive all the heartbeat information (heartbeat) of the slave node (TT) and requests from clients. JT assumes a lot of responsibilities. As the cluster expands, the pressure on JT will become greater and greater.

Finally, the biggest problem is compatibility, which is not compatible with frameworks other than MapReduce, such as Spark, which cannot run on this system.

With YARN, many other computing frameworks can be run based on YARN, and different computing frameworks can share data on the same HDFS cluster and enjoy the overall resource scheduling. It is equivalent to the operating system and plays the role of scheduling management.

Overview of YARN

The full name of YARN is Yet Another Resource Negotiator.

General resource management system, to apply for resources unified through the YARN application on the line.

Provide unified resource management and scheduling for upper-level applications.

The architecture of YARN is shown in the following figure:

The architecture of YARN consists of these parts:

ResourceManager (RM): resource Manager

There is only one RM in the whole cluster that provides services at the same time, which is responsible for the unified management and scheduling of cluster resources. Generally speaking, there is only one RM, and it is easy to be confused if there is more than one, but if there is only one, if there is a problem with the RM, the whole system will collapse, so another one will be added as a backup as far as possible in production, so that even if the primary RM is down, the standby RM can continue to work, but there is only one providing service at the same time, either the primary RM or the standby RM.

Process client requests: submit jobs, kill jobs.

Monitor the NM, and once a NM is down, the task running on that NM needs to tell AM how to handle it.

NodeManager (NM): node Manager

There are multiple NM in the whole cluster, which are responsible for the management and use of their own node resources. NM can know to manage the resources of its own nodes, such as how many resources are left at a certain time.

Regularly report the resource usage of this node to RM. RM can schedule reasonably only if it knows the resource usage on all NM. For a particular assignment, he knows which NM to assign to.

Accept and process various commands from RM, such as starting Container. Since NM is the slave node of the master-slave structure, you should listen to the boss. The boss assigned you a task and asked you to execute it. If you don't do it, don't do it.

Processing commands from AM, AM tells NM how many container runs task,NM needs to be started on the node to run.

In the resource management of a single node, in the process of running jobs, there must be a number of resources on their own nodes and how many resources are left.

ApplicationMaster (AM): application master

There is one for each application, such as a Spark or a MapReduce job for an AM. Responsible for the management of the application, request resources from RM for the application, such as how much memory and computation are required. After getting the resources, they are allocated to the internal task for processing.

Need to communicate with NM, start or stop task,task is running in container, AM is also running in container.

Container: container

A container that encapsulates resources such as CPU, Memory, etc., which is equivalent to the abstraction of a task running environment.

Client: client:

Submit a job, check the progress of the job, and kill the job

This is what I understand about this architecture, which can be compared with the management of an enterprise or company:

Client is simply a customer who cooperates with the company. Submitting an assignment is equivalent to a deal between the customer and the company, the customer makes a request, and the company is responsible for helping the customer complete the project, but there is no money transaction here.

RM, the equivalent of the boss in charge of the company, is so powerful that all matters of the company have to be obeyed by him. We compare the role of RM to understand, first of all, a company can only have one decision at a time, you can not say yours, he said his, and finally the whole company does not know who to listen to, then it is completely out of order. If one day the boss is ill, he can only entrust an agent to temporarily help him exercise the management responsibility, this agent is the backup RM. Second, it is understandable that the boss can decide when to terminate the project and when to start. NM is equivalent to many departments under the company, and AM is equivalent to the person in charge of a specific project. We'll talk about it later, so we'll understand it this way for the time being. Finally, the boss can know the situation of the department at any time. If a department has a collective holiday or some major accident, then naturally the project involved in this department will be shelved. At this time, how to deal with this project? should we wait for them to come back to continue, or directly give the project to other departments? this is to ask the person in charge of the project, how to decide to listen to his opinion. After all, he is in charge of the project.

The whole node where NM is located is equivalent to each department of the company. the only difference is that in the company, there are often not two financial departments or three human resources departments, and the company's department is often the only one. In the structure of YARN, each node is not unique, so for the sake of analogy, we can assume that the company has three development departments and four test departments. In this way, NM can be understood as the person in charge of this department. Obviously, NM knows a lot about the situation of this department, including who has come to work today, what is the current working ability of the department, whether someone is still in the state of having nothing to do, and so on. The person in charge needs to submit work records (heartbeat information) to the boss regularly to let the boss know the ability of the department, and so on, so that the boss will know if he can be assigned to the department the next time there is a new task. As the head of the department, you naturally have to listen to the boss. The boss gives you a task and asks you to set up a container in the department immediately. You have to carry out the boss's orders. At the same time, he should also try to meet the requirements of the project leader as much as possible. for example, the project leader said which people in your department needed to do it for me in this project. The head of the department formed those people into a group to carry out the task in accordance with his request.

AM, project leader or team, a company may have multiple projects, naturally each project needs to have a project leader. The person in charge of the project is bound to use the resources of the company when doing the project, such as meeting rooms and printers for meetings, so naturally he has to apply to the boss that my project needs to use the company's conference rooms, printers and so on. The boss says yes, ah, no problem, so after he gets these resources, he will say to each team, I have applied for these things for you. You can use it directly if you want to use it in the future. Finally, he can say to the head of a department, I need several people in your department to form a container to help me with this project, or this group in your department can't do it well, I don't want them.

Container is a group, in which there are some resources that everyone can share, and each project is divided into small tasks that are completed in each group. Each team belongs to a certain department, and a department can have several groups.

I don't know if this will help to understand the architecture of YARN, but this is just some of my understanding when I saw the architecture.

YRAN execution process

The execution process of YARN

The client submits a job request to RM, either a MapReduce job or a Spark job.

RM will assign the first container to the job, assuming that the container is running on the second node, so that RM will communicate with the corresponding NM, that is, say to the NM of the second node, I'm going to start a container on you.

After receiving the instruction from RM, NM launches a container on NM, and application master runs in this container.

After AM starts, it registers in RM, and the registered user can see the progress of the job execution through RM. And AM will apply to RM for the resources you need, such as how much memory you need, and enjoy yourself if you apply for the resources, and then proceed to the following steps.

After applying for the resource, AM starts the task on the corresponding NM. Suppose you need to start two task on the first NM and one task on the third NM, then send these notifications to the corresponding NM.

After receiving these notifications, NM knows how many task it needs to create, so launch the corresponding container on the NM and put the task into the container to run.

In fact, this process is not very complex, you can compare with the previous example of the company, bring the corresponding role into the understanding of the process.

Let's think about another question: why can't other computing frameworks be run in version 1.x, but only after using YARN? The key is that this process is a general process, AM as the main control program of the application, if we make the corresponding AM implementation for the corresponding framework, that is to say, if it is MapReduce, then the AM here is the AM corresponding to MapReduce, and the same is true for spark. Then you can run many computing frameworks on top of YARN. In fact, the role of YARN can be understood as an operating system that can run a variety of computing frameworks, just like using the Windows operating system, you can run all kinds of software on this operating system.

This is the end of the content of "what are the knowledge points related to YARN". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.