Shulou(Shulou.com)06/03 Report--

A brief introduction to Yarn

We know that in the field of offline big data processing, hadoop is the current understandable processing architecture. So far, there are three major versions of hadoop, each of which has architectural adjustments.

There are some disadvantages in hadoop1.0, such as the single-node failure of saving metadata information in hadoop1.0, and the task computing framework can only use mapreduce, which causes too much pressure on the task manager, so the mechanism of unified management of yarn resources is added to hadoop2.0, which not only solves the problem of metadata single-node failure (double namenode), but also realizes the real-time hot backup of metadata (JournalNode), and adds yarn between hdfs and mr. Unify and coordinate resources.

In this article, we only introduce the operation principle of yarn, and you can query the relevant documents for other related knowledge, so I won't introduce it here.

Schematic diagram of Yarn (screenshot here)

Introduction of principle 1. The client executes the run method, starts the task, and detects the corresponding permission request path during startup. If the detection fails, the subsequent execution is terminated.

two。 After the first step has passed, send the request to resourcemanager and return the task id and the storage strength of the task in hdfs.

3. After receiving the response information, the client packages and uploads the job task to hdfs (shared file system) according to the submission path.

4. The client submits the job task (application) to resourcemanager again

5. Resourcemanager applies for a resource container container according to the task submission, and the applicationmanager selects a nodemanager (the node's resource manager), launches a container of the same size as the application, and opens an applicationmaster (the sub-instance coordinator of the task)

6. Appalicationmaster initializes the task

7. Get the location of the executing task in the hdfs from resourcemanager, and calculate the number of mapreduce based on the task information

8. According to the initialization and calculation results, Appalicationmaster then goes to resourcemanager to apply for the container required by the task (mapreduce) task according to the size of the mapreduce task.

9. After Applicationmaster applies for container, select a nodemanager again and send the instruction to start container.

10. After Nodemanager receives the instruction, start container and get the executed mapreduce task from the hdfs road strength where the task is located.

11. Start the jvm virtual machine to perform the acquired mapreduce task

Note: applicationmaster will monitor the execution of each nodemanager and report to resourcemanager at any time, until the final task execution is completed, all resources will be reclaimed. If a task is found to fail, applicationmaster will coordinate it, reducing the burden on resourcemanager. If there is a problem with applicationmaster during operation, resourcemanager will not receive the heartbeat, then it will select one of all nodemanager nodes to start applicationmaster to continue to work.

That's how yarn works in hadoop2.0. We learn from each other-- growth starts with blogs.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.