Shulou(Shulou.com)06/02 Report--

This article mainly explains "what YARN is". The explanation in this article is simple and clear, easy to learn and understand. Please follow the ideas of Xiaobian and go deep into it slowly to study and learn "what YARN is" together.

YARN is a new Hadoop resource manager. It is a general resource management system that provides unified resource management and scheduling for upper applications. Its introduction brings great benefits to clusters in terms of utilization, unified resource management and data sharing.

YARN is still master/slave structure in general, resource manager is master and node manager is slave in the whole resource management framework. Resourcesmanager is responsible for unified management and scheduling of resources on each nademander. When a user submits an application, an ApplicationMaster is provided to track and manage the application, which is responsible for requesting resources from the ResourceManager and asking NodeManger to start tasks that can consume certain resources. Since different ApplicationMasters are distributed to different nodes, they do not affect each other.

YARN consists of ResourceManager, NodeManager, ApplicationMaster and Container.

ResourceManager is an independent process running on Master, responsible for unified resource management, scheduling, allocation, etc. of cluster;NodeManager is an independent process running on Slave, responsible for reporting node status;App Master and Container are components running on Slave, Container is a unit of resource allocation in Yarn, including memory, CPU, etc. Yarn allocates resources in units of Container. Each application submitted by Client to ResourceManager must have an Application Master, which runs in a Container of a Slave node after resources are allocated by ResourceManager, and the Task that does specific things also runs in a Container of a Slave node. Communication between RM, NM, AM and even ordinary containers uses RPC mechanism.

YARN is architected to look more and more like a cloud operating system, a data processing operating system.

Finally, YARN's resource management can be understood from the following aspects:

1. Resource scheduling and isolation are two of the most important and fundamental functions of Yarn as a resource management system. Resource scheduling is done by the resource manager, and resource isolation is done by the node managers.

2. After the Resourcemanager allocates resources on a node manager to a task (this is the so-called "resource scheduling"), the node manager needs to provide corresponding resources for the task according to the requirements, and even ensure that these resources should be exclusive, providing a basis and guarantee for the task operation. This is the so-called resource isolation.

3. When we talk about resources, we usually refer to memory, cpu and io. Hadoop Yarn currently only supports CPU and memory resource management and scheduling.

4. Memory resources determine the life and death of a task. If memory is insufficient, the task may fail. In contrast, cpu resources are different. They only determine the speed of the task and will not affect the life and death of the task.

Thank you for reading, the above is the content of "YARN is what", after the study of this article, I believe everyone has a deeper understanding of what YARN is, the specific use of the situation still needs to be verified by practice. Here is, Xiaobian will push more articles related to knowledge points for everyone, welcome to pay attention!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.