Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "how to solve the problem that CDH cluster installation YARN can not start normally". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

1 the purpose of document writing

When installing the version 5.11.2 CDH cluster, or expanding the capacity of the version 5.11.2 CDH cluster, you may encounter the problem that the NodeManager instance of YARN cannot be started. This article mainly describes the reason why NodeManager cannot be started and how to solve the problem.

Test environment:

The operating system version is Redhat7.2

Version 5.11.2 for CM and CDH

Article directory structure:

1. Purpose of document writing

two。 Problem description

3. Solution

3.1 problem solving

3.2 relevant recommendations

4. Summary

2 problem description

When using the CDH5.11.2 version, the NodeManager role of YARN is added, and the following exception occurs when starting the role instance:

Check the exception log of YARN. The exception information is as follows:

July 16, 10: 58 p.m. 42.862 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager

Error starting NodeManager

Org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: / var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm-state/LOCK: Permission denied

At org.apache.hadoop.service.ServiceStateException.convert (ServiceStateException.java:59)

At org.apache.hadoop.service.AbstractService.init (AbstractService.java:172)

At org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore (NodeManager.java:181)

At org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit (NodeManager.java:245)

At org.apache.hadoop.service.AbstractService.init (AbstractService.java:163)

At org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager (NodeManager.java:562)

At org.apache.hadoop.yarn.server.nodemanager.NodeManager.main (NodeManager.java:609)

Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: / var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm-state/LOCK: Permission denied

At org.fusesource.leveldbjni.internal.NativeDB.checkStatus (NativeDB.java:200)

At org.fusesource.leveldbjni.internal.NativeDB.open (NativeDB.java:218)

At org.fusesource.leveldbjni.JniDBFactory.open (JniDBFactory.java:168)

At org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.openDatabase (NMLeveldbStateStoreService.java:944)

At org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage (NMLeveldbStateStoreService.java:931)

At org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit (NMStateStoreService.java:204)

At org.apache.hadoop.service.AbstractService.init (AbstractService.java:163)

... 5 more

I have encountered this kind of problem when I deployed the 5.11.2 version of the CDH cluster on the Redhat7.2 operating system and expanded the nodes for this version of the cluster. The solution to the problem is relatively simple. The main purpose of writing this article is to provide a reference for avoiding such problems in the future.

3 solution 3.1 problem solving steps

According to YARN's exception log, you can see that the prompt message is "IO error:/var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm-state/LOCK: Permission denied". You need to log in to NodeManager to troubleshoot nodes with abnormal startup.

1. Log in to the bigdata29 node, check the / var/lib/hadoop-yarn/yarn-nm-recovery/ directory, and find that the directory is empty

2. Check the relevant directory permissions and find that the hadoop-yarn directory permission is 000. locate the problem.

Root@bigdata29:/var/lib/hadoop-yarn > ls-lrt / var/lib/ | grep hadoop-ya

D Murray. 3 root root 4096 Jul 16 22:39 hadoop-yarn

3. Change the permissions of the hadoop-yarn directory to 755

4. Restart the NodeManager role instance of bigdata29 and start it normally.

3.2 relevant recommendations

This problem can be avoided by manually creating the / var/lib/hadoop-yarn directory on the relevant nodes before adding NodeManager. If there are too many nodes, you can create a directory by bulk executing the command script.

4 Summary

When using the CDH5.11.2 version, the new NodeManager role of YARN may cause the / var/lib/hadoop-yarn/ directory permission to be 000. when this problem occurs, you need to change the directory permission to 755, and then restart NodeManager. To avoid this problem, you can create / var/lib/hadoop-yarn/ in advance.

This is the end of the content of "how to solve the problem that the CDH cluster installation YARN cannot be started properly". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.