Shulou(Shulou.com)05/31 Report--

This article mainly introduces "what are the new features of Hadoop YARN". In daily operation, I believe many people have doubts about the new features of Hadoop YARN. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful for you to answer the doubts about "what are the new features of Hadoop YARN?" Next, please follow the editor to study!

(1) ResourceManager HA

After apache hadoop 2.4 or CDH5.0.0 version, ResourceManger HA feature has been added to support hot master / slave switching based on Zookeeper. For more information on configuration parameters, please see Cloudera document: ResourceManager HA configuration.

It is important to note that ResourceManager HA has only completed the first phase of the design, that is, after the ResourceManager starts, it kills the previously running Application, then reads the Application's metadata information from the shared storage system, and resubmits the Application. After starting ApplicationMaster, the remaining fault tolerance functions are left to ApplicationMaster to implement. For example, MapReduce's ApplicationMaster will constantly write completed task information to HDFS, so that when it restarts, it can reread these logs, and then only need to rerun those unfinished tasks. The task of the second phase of ResourceManager HA is that after the ResourceManager takes over the master ResourceManager, you don't have to kill those running Application and let them run as if nothing had happened.

(2) disk fault tolerance

After apache hadoop 2.4 or CDH5.0.0, several disk-friendly parameters have been added, which allow YARN to make better use of multiple disks on NodeManager. The related jira is: YARN-1781, with three new parameters:

The healthy disk ratio is at least guaranteed on yarn.nodemanager.disk-health-checker.min-healthy-disks:NodeManager. When the healthy disk ratio is lower than this value, NodeManager will no longer receive and start new Container. The default value is 0.25, which means 25%.

Yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage: the maximum utilization rate of a disk. When the utilization rate of a disk exceeds this value, the disk is considered to be a bad disk and the disk is no longer used. The default is 100, which means 100%. It can be adjusted down appropriately.

Yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb: a disk ensures at least the amount of remaining space. When the remaining space of a disk is lower than this value, the disk will no longer be used. The default is 0, which means 0MB.

(3) Resource scheduler

Fair Scheduler:Fair Scheduler adds a very useful new feature that allows users to transfer an application online from one queue to another, such as moving an important job from a low-priority queue to a high-priority queue. The operation command is: bin/yarn application-movetoqueue appID-queue targetQueueName, and the related jira is: YARN-1721.

The resource preemption feature in Capacity Scheduler:Capacity Scheduler has been fully tested and ready for use.

At this point, the study on "what are the new features of Hadoop YARN" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.