Shulou(Shulou.com)05/31 Report--

This article introduces the relevant knowledge of "how to solve the problems encountered in the use of hadoop+hive". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

1.datanode failed to start normally

After adding datanode, datanode cannot start normally, and the process will hang up inexplicably. Check the namenode log as shown below:

2013-06-21 1815 53 FATAL org.apache.hadoop.hdfs.StateChange 39182: BLOCK* NameSystem.getDatanode: Datanode x.x.x.x:50010 is attempting to report storage ID DS-1357535176-x.x.x.x-50010-1371808472808. Node y.y.y.y:50010 is expected to serve this storage.

Cause analysis:

When copying the hadoop installation package, it contains the data and tmp folders, and datanode is not formatted successfully

Solution:

Rm-rf / data/hadoop/hadoop-1.1.2/datarm-rf / data/hadoop/hadoop-1.1.2/tmphadoop datanode-format

2. Safe mode

2013-06-20 10 Cannot renew lease for DFSClient_hb_rs_wdev1.corp.qihoo.net,60020,1371631589073 3543758 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hadoop cause:org.apache.hadoop.hdfs.server.namenode.SafeModeException. Name node is in safe mode.

Solution:

Hadoop dfsadmin-safemode leave

3. Connection exception

2013-06-21 19 Call to homename/x.x.x.x:9000 failed on local exception 5515 05801 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call to homename/x.x.x.x:9000 failed on local exception: java.io.EOFException

Possible reasons:

Namenode snoops 127.0.0.1 9000 instead of 0.0.0.0 9000 or public network limit

Solution:

Check the / etc/hosts configuration so that hostname is bound to the iptables release port on IP other than 127.0.0.1

4. Namenode id

ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in / var/lib/hadoop-0.20/cache/hdfs/dfs/data: namenode namespaceID = 240012870; datanode namespaceID = 1462711424.

Problem: namespaceID on Namenode is inconsistent with namespaceID on datanode.

The cause of the problem: each time the namenode format recreates a namenodeId, and the id,namenode format under the tmp/dfs/data includes the id,namenode format under the last format to empty the data under the namenode, but not the data under the datanode, which causes the namespaceID on the namenode node to be inconsistent with the namespaceID on the datanode node. Failed to start.

Solution: refer to the website http://blog.csdn.net/wh72592855/archive/2010/07/21/5752199.aspx to give two solutions. We use the first solution: namely:

(1) stop the cluster service

(2) Delete the data directory on the datanode node where the problem occurs. The data directory is the dfs.data.dir directory configured in the hdfs-site.xml file, and the one on this machine is / var/lib/hadoop-0.20/cache/hdfs/dfs/data/ (Note: we performed this step on all datanode and namenode nodes at that time. In case of unsuccessful deletion, you can save a copy of the data directory first.

(3) format namenode.

(4) restart the cluster.

Problem solved.

One side effect of this approach is that all data on the hdfs is lost. If this method is not recommended when there is important data on the hdfs, you can try the second method in the URL provided.

5. Directory permissions

Start-dfs.sh execution is error-free, showing startup datanode, and no datanode after execution. Check the log on the datanode machine and show that it is caused by incorrect permissions of the dfs.data.dir directory:

Expected: drwxr-xr-x,current:drwxrwxr-x

Solution:

Check the directory configuration of dfs.data.dir and modify the permissions.

Hive error

1.NoClassDefFoundError

Could not initialize class java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.io.HbaseObjectWritable

Add protobuf-***.jar to the jars path

/ / $HIVE_HOME/conf/hive-site.xml hive.aux.jars.path file:///data/hadoop/hive-0.10.0/lib/hive-hbase-handler-0.10.0.jar,file:///data/hadoop/hive-0.10.0/lib/hbase-0.94.8.jar,file:///data/hadoop/hive-0.10.0/lib/zookeeper-3.4.5.jar, File:///data/hadoop/hive-0.10.0/lib/guava-r09.jar,file:///data/hadoop/hive-0.10.0/lib/hive-contrib-0.10.0.jar,file:///data/hadoop/hive-0.10.0/lib/protobuf-java-2.4.0a.jar

2.hive dynamic partitioning exception

[Fatal Error] Operator FS_2 (id=2): Number of dynamic partitions exceeded hive.exec.max.dynamic.partitions.pernode

Hive > set hive.exec.max.dynamic.partitions.pernode = 10000

3.mapreduce process exceeds memory limit-- hadoop Java heap space

Vim mapred-site.xml add:

/ / mapred-site.xmlmapred.child.java.opts-Xmx2048m

# $HADOOP_HOME/conf/hadoop_env.shexport HADOOP_HEAPSIZE=5000

Limit the number of 4.hive files

[Fatal Error] total number of created files now is 100086, which exceeds 100000

Hive > set hive.exec.max.created.files=655350

5.metastore connection timed out

FAILED: SemanticException org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out

Solution:

Hive > set hive.metastore.client.socket.timeout=500

6. Java.io.IOException: error=7, Argument list too long

Task with the most failures (5):-Task ID: task_201306241630_0189_r_000009URL: http://namenode.godlovesdog.com:50030/taskdetails.jsp?jobid=job_201306241630_0189&tipid=task_201306241630_0189_r_000009-----Diagnostic Messages for this Task:java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key": {"reducesinkkey0": "164058872", "reducesinkkey1": "djh,S1" "reducesinkkey2": "20130117170703", "reducesinkkey3": "xxx"}, "value": {"_ col0": "1", "_ col1": "xxx", "_ col2": "201301170703", "_ col3": "164058872", "_ col4": "xxx,S1"} "alias": 0} at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce (ExecReducer.java:270) at org.apache.hadoop.mapred.ReduceTask.runOldReducer (ReduceTask.java:520) at org.apache.hadoop.mapred.ReduceTask.run (ReduceTask.java:421) at org.apache.hadoop.mapred.Child$4.run (Child.java:255) at java.security.AccessController.doPrivileged (Native Method) at javax.security.auth.Subject.doAs (Subject.java:415) at org.apache. Hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java:1149) at org.apache.hadoop.mapred.Child.main (Child.java:249) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key": {"reducesinkkey0": "164058872" "reducesinkkey1": "xxx,S1", "reducesinkkey2": "20130117170703", "reducesinkkey3": "xxx"}, "value": {"_ col0": "1", "_ col1": "xxx", "_ col2": "20130117170703", "_ col3": "164058872", "_ col4": "djh,S1"}, "alias": 0} at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce (ExecReducer.java:258). 7 moreCaused by: org.apache.hadoop.hive.ql.metadata.HiveException: [Error 20000]: Unable to initialize custom script.at org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp (ScriptOperator.java:354) at org.apache.hadoop.hive.ql.exec.Operator.process (Operator.java:474) at org.apache.hadoop.hive.ql.exec.Operator.forward (Operator.java:800) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp (SelectOperator.java: 84) at org.apache.hadoop.hive.ql.exec.Operator.process (Operator.java:474) at org.apache.hadoop.hive.ql.exec.Operator.forward (Operator.java:800) at org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp (ExtractOperator.java:45) at org.apache.hadoop.hive.ql.exec.Operator.process (Operator.java:474) at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce (ExecReducer.java:249). 7 moreCaused by: java.io.IOException: Cannot run program "/ usr/bin/python2.7": error=7, parameter list is too long at java.lang.ProcessBuilder.start (ProcessBuilder.java:1042) at org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp (ScriptOperator.java:313). 15 moreCaused by: java.io.IOException: error=7, the parameter list is too long at java.lang.UNIXProcess.forkAndExec (Native Method) at java.lang.UNIXProcess. (UNIXProcess.java:135) at java.lang.ProcessImpl.start (ProcessImpl.java:130) at java.lang.ProcessBuilder.start (ProcessBuilder.java:1023). 16 moreFAILED: Execution Error, return code 20000 from org.apache.hadoop.hive.ql.exec.MapRedTask. Unable to initialize custom script.

Solution:

Upgrade the kernel or reduce the number of partitions

This is the end of the content of "how to solve the problems encountered in the use of hadoop+hive". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.