Shulou(Shulou.com)06/03 Report--

The basic theory of 1.YARN 1) introduction to YARN:

   YARN is a resource scheduling platform, which is responsible for providing server computing resources for computing programs, which is equivalent to a distributed operating system platform, while computing programs such as MapReduce are equivalent to applications running on the operating system.

2) deficiency of YARN in hadoop1.x:

  -JobTracker is the centralized processing of cluster transactions, with a single point of failure.

  -JobTracker needs too many tasks to maintain both the state of job and the state of task of job, resulting in excessive consumption of resources

  -on the TaskTracker side, the representation of Map/Reduce Task as a resource is too simple to take into account CPU. Memory, and other resources, the two need to consume a large amount of Task dispatched together, it is easy to appear OOM.

  -the resource is forcibly divided into Map/Reduce Slot. When there is only MapTask, TeduceSlot cannot be used; when there is only ReduceTask, MapSlot cannot be used, which can easily lead to underutilization of resources.

3) New features of YARN in hadoop2.x:

The most basic idea of    MRv2 is to separate the main resource management and Job scheduling / monitoring functions of the original JobTracker as two separate daemons. There is a global ResourceManager (RM) and each Application has an ApplicationMaster (AM), and Application is equivalent to MapReduce Job or DAG jobs. ResourceManager and NodeManager (NM) constitute the basic data computing framework. ResourceManager coordinates the resource utilization of the cluster. Any Client or running applicatitonMaster that wants to run Job or Task must apply for certain resources from RM. ApplicatonMaster is a framework-specific library that has its own AM implementation for the MapReduce framework.

Users can also implement their own AM, and at run time, AM works with NM to launch and monitor Tasks.

4) introduction of roles in YARN:

ResourceManager:ResoueceMananer is the main control node of yarn cluster, which is based on the needs of applications to schedule cluster resources. It is responsible for coordinating and managing the whole cluster, and different types of applications submitted by corresponding users, such as parsing, scheduling, monitoring and so on. ResourceManager launches a MRappmaster for each application, and the MRappmaster is scattered across the nodemanager.

As long as ResourceManager consists of two parts:

  -Application Manager (ApplicationsManager, ASM): manages and monitors the MRappmaster of all applications, the MRappmaster that starts the application, and the MRappmaster failed restart

  -scheduler (Scheduler): the underlying queue is responsible for the execution time and order of the application

    -FIFO (first-in, first-out queue): only one queue is maintained internally for tasks that are submitted first and then submitted.

    -Fair Fair Scheduler: all computing tasks are divided equally. If there is only one job in the global, then the current job takes up all resources.

    -Capacity (Computing Power Scheduler): resources can be configured according to the size of the actual job task

NodeManager:Nodemanager is the provider of real resources in the yarn cluster and the provider of containers that actually execute the application, monitoring the resources of the application (cpu, network, IO, memory). And through the heartbeat to the cluster master node ResourceManager to report and update their health status. At the same time, it will also supervise the life cycle management of container and monitor the resources of each container.

MRAppMaster: request resources from ResourceManager for mapTask and reduceTask of current job, monitor the operation and progress of mapTask and reduceTask of current job, restart failed MapTask and reduceTask, and be responsible for resource recovery of mapTask and reduceTask.

Container: Container is a container, an abstract logical resource unit. The container is composed of resources dynamically allocated by the ResourceManager Scheduler service, which includes a certain amount of cpu, network, IO and memory on the node. All the Task of the MapReduce program is executed in a container.

5) Resource scheduling in YARN:

Hadoop 1.x:

First of all, the computing job is initiated through client. JobTracker initiates computing to each Task Tracker (each jobTracker is a map-reduce). The specific work is Task Tracker, while jobTracker does resource management. First, Task Tracker makes a heartbeat to job tracker, and pulls the corresponding task at the same time to start computing. First, job Tracker controls Task Tracker parallel map computing, and when map ends, job Tracker controls Task Tracker parallel reduce computing.

Hadoop 2.x:

When the client submits a computing task to resourceManager (hadoopxx.jar) resourceManager, it launches a container on a node, in which a MRappmasterMRappmaster is run to apply for resources from resourceManager, maptask and reducetaskresourceManager return to MRAPPmaster the nodes running maptask and reducetask MRAPPmaster to the corresponding node to launch a container in which maptask and reudcetaskMRappmaster are running to monitor the health status of maptask or reducetask. After running maptask or reducetask, it applies to MRappmaster to log off itself, and releases resources MRappmaster to resourcemanager to log off itself and release resources. Job submission process for 2.YARN

The client submits the job running request (hadoop jar xxxx.jar) Resourcemanager to resourcemanager for inspection. When there is no problem, it returns a shared resource path to the client and the JobID client puts the shared resources under the shared path: (/ tmp/hadoop-yarn/staging/hadoop/.staging/job_1539740094604_0002/) the jar package that Job.jar needs to run Renamed to job.jarJob.split slice Information (FlieInputFormat---getSplits List) Job.xml profile Information (some columns of job.setxxxx ()) the client reports to resourcemanager that the shared resource has been placed Make the real submission of job resourceManager assign a node to the job and start the MRAPPmaster task resourceManager on this node to start a container and then start mrappmasterMRappmaster to download resources (mainly split, job) from the resource path to initialize job to generate an job workbook. The job workbook records the progress and status of maptask and reduce. MRappmaster applies to resourcemanager for resources to run maptask and reducetask. Send maptask first and then send reducetaskresourcemanager to MRAPPmaster to return the resource nodes of maptask and reduce (when returning the node, there is a principle of proximity. Priority is given to returning the actual node of the slice processed by the current maptask. When the data is processed, the data can be localized. If it is multiple copies, it is on any node of multiple copies. The reducetask task starts on any node that is not busy) MRAPPmaster starts a container on the corresponding node, then starts the maptask task maptask task in container to download the corresponding resource (running jar package) maptask task in the corresponding shared resource path, and periodically reports its running status and progress to MRAPPmaster. When a maptask task is completed, reduce starts container and then starts the reduce task. But the reducetask here only does the work of data pull, and does not calculate the duceTask task to the corresponding shared resource path to download the corresponding resources (running jar package). When all maptask tasks are completed, start the reduce task to calculate. When the maptask or reducetask task is completed, it will apply to MRAPPmaster to log off itself and release resources. When the application task is completed, MRAPPmaster will apply to resourcemanager to log off itself and release resources.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.