Shulou(Shulou.com)06/03 Report--

What is the Hadoop cluster resource manager? How does it provide resource management and scheduling for upper-level applications? Let's take a look at the huge benefits that the introduction of Apache Hadoop YARN brings to the cluster in terms of utilization, unified management of resources and data sharing.

From the developer's point of view, the job (application) is submitted to the YARN cluster, and the job is monitored and managed through YARN's UI. Hadoop provides developers with three main YARN tools: (1) Hue Job Browser; (2) YARN Web UI; (3) YARN command. YARN administrators can use Cloudera Manager, which is included in Cloudera Express and ClouderaEnterprise, and is also of some help to developers. Let's take a look at the role of specific components:

(1) Hue Job Browser: you can monitor job status, view logs, and kill jobs

(2) YARN Web UI:Resource Manager UI is the main entry point, running on port 8088 of the RM host. Compared with Hue, it provides more detailed views (you can see the tasks under each application, how many attempt are in the tasks, and some logs), but it does not provide any control or configuration. Here we take a look at the categories of YARN Web UI:

1) Resource Manager UI:Nodes

2) Resource Manager UI:Applications

3) ResourceManager UI:Application Detail

(3) YARN command line

1) configure and view information about YARN clusters through commands

A) yarn

2) most YARN command line tools are for administrators rather than developers

3) some commands that are useful to developers

B) yarn application, use-list to view running applications, and use-kill to kill running applications.

C) yarn logs-applicationId to view the log of a specific application

(4) Cloudera Manager

Cloudera Manager provides a powerful ability to monitor and configure clusters from a single node

As the resource manager of Hadoop, YARN brings convenience to the resource management and sharing of the cluster, but we must make clear the characteristics of YARN in practical application in order to better grasp and apply. Due to the needs of current development, big data has provided opportunities and challenges for everyone who wants to engage in or has already engaged in big data. We should not only think about and sum up in many directions, but also learn from and learn from the experiences of others. Only by constantly improving our own knowledge structure can we make greater progress. You can usually follow the Wechat information platform such as "big data cn". The relevant knowledge shared above is very novel and unique for us practitioners, and it is very helpful to broaden our horizons, expand our knowledge and stimulate our thinking.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.