Shulou(Shulou.com)06/01 Report--

This article introduces the knowledge of "how to solve errors after adding new datanode nodes in hadoop2.7 cluster". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Our hadoop is 2.7 version, get the latest version at that time, alas, unstable ah, a lot of twists and turns in the middle.

The current cluster is 1 namenode,4 per datanode. The server is a virtual machine, the configuration is too poor, need to add nodes.

For the miserable hadoop cluster, I added 3 datanode nodes at the same time. It turned out that the cluster had 4 data nodes, probably because too many data nodes were added, and the server resources were quickly exhausted when balancer was executed.

Check that the maximum number of processes set on the server is 1024, and balancer will start the native process when it is executed. Because there are too many balance operations to be performed, the cluster will constantly start local processes and exhaust the number of processes. Even the ps command didn't work, but luckily I wrote down the process number of balancer and finally dropped kill. The error message log is as follows:

2015-09-11 20 DFSClient_NONMAPREDUCE_880883068_1 04 DFSClient_NONMAPREDUCE_880883068_1 11175 WARN org.apache.hadoop.hdfs.LeaseRenewer: Failed to renew lease for [DFSClient_NONMAPREDUCE_880883068_1] for 73 seconds. Will retry shortly...

Java.io.IOException: com.google.protobuf.ServiceException: java.lang.OutOfMemoryError: unable to create new native thread

At org.apache.hadoop.ipc.ProtobufHelper.getRemoteException (ProtobufHelper.java:47)

At org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease (ClientNamenodeProtocolTranslatorPB.java:592)

At sun.reflect.GeneratedMethodAccessor2.invoke (Unknown Source)

At sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)

At java.lang.reflect.Method.invoke (Method.java:606)

At org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod (RetryInvocationHandler.java:187)

At org.apache.hadoop.io.retry.RetryInvocationHandler.invoke (RetryInvocationHandler.java:102)

At com.sun.proxy.$Proxy12.renewLease (Unknown Source)

At org.apache.hadoop.hdfs.DFSClient.renewLease (DFSClient.java:891)

At org.apache.hadoop.hdfs.LeaseRenewer.renew (LeaseRenewer.java:417)

At org.apache.hadoop.hdfs.LeaseRenewer.run (LeaseRenewer.java:442)

At org.apache.hadoop.hdfs.LeaseRenewer.access$700 (LeaseRenewer.java:71)

At org.apache.hadoop.hdfs.LeaseRenewer$1.run (LeaseRenewer.java:298)

At java.lang.Thread.run (Thread.java:745)

Caused by: com.google.protobuf.ServiceException: java.lang.OutOfMemoryError: unable to create new native thread

At org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke (ProtobufRpcEngine.java:243)

At com.sun.proxy.$Proxy11.renewLease (Unknown Source)

At org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease (ClientNamenodeProtocolTranslatorPB.java:590)

... 12 more

Caused by: java.lang.OutOfMemoryError: unable to create new native thread

At java.lang.Thread.start0 (Native Method)

At java.lang.Thread.start (Thread.java:714)

At java.util.concurrent.ThreadPoolExecutor.addWorker (ThreadPoolExecutor.java:949)

At java.util.concurrent.ThreadPoolExecutor.execute (ThreadPoolExecutor.java:1371)

At java.util.concurrent.AbstractExecutorService.submit (AbstractExecutorService.java:110)

At org.apache.hadoop.ipc.Client$Connection.sendRpcRequest (Client.java:1022)

At org.apache.hadoop.ipc.Client.call (Client.java:1449)

At org.apache.hadoop.ipc.Client.call (Client.java:1407)

At org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke (ProtobufRpcEngine.java:229)

... 14 more

Idea 1: try to adjust the threshold of balancer. According to the Internet, the value is 0,100. According to the order of 90, 80, 70, 50, 30, 10, this little bit of balance will not be better. After actual use, it is found that all those greater than 10 will not work, so this method will not work. Failure!

Idea 2: whether it is because of the sudden addition of three nodes, the background balancer of hadoop has too much content (the current amount of data is 370G, and most of them are hbase files). Would it be better for me to increase one node at a time? As a result, I removed the nodes of two of the three, and only one was left to perform balancer. Well, at first I thought it would work, but after running for about a few minutes, the same problem came up again. Failure!

Later found that the new host and the original hadoop host time zone setting is not the same, changed to China's Shanghai time zone, or the following problems. However, the time zone setting must be set, which can avoid another hidden problem in advance.

Idea 3: because the server uses vm, the configuration is not very high, so I do not want to change the system parameters to tune, if the parameter value is increased, the use efficiency of linux will increase, at the same time, the vm host may not be able to bear, resulting in system downtime. But there's nothing you can do about it. Execute ulimit-an and check "max user processes (- u) 1024" in the result.

Execute vi / etc/security/limits.d/90-nproc.conf, change 1024 to 5120, and save and exit.

# Default limit for number of user's processes to prevent

# accidental fork bombs.

# See rhbz # 432903 for reasoning.

* soft nproc 1024

Root soft nproc unlimited

Then execute start-balancer.sh threshold 5, and don't make a mistake this time. The end can be executed normally.

This is the end of the content of "how to solve errors after adding datanode nodes in hadoop2.7 cluster". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.