Shulou(Shulou.com)05/31 Report--

This article mainly shows you "what are the common problems in Hive". The content is simple and clear. I hope it can help you solve your doubts. Let the editor lead you to study and learn this article "what are the common problems in Hive?"

Hive single table has too many partitions to execute query error (in fact, the more partitions, the slower the query. You should control the number of partitions below 5000):

Java.lang.OutOfMemoryError: Java heap space

Reference: OOM occurs when query spans to a large number of partitions

Reason:

When executing the query, hive will first load the partition information in the metadata into memory, including the data of PARTITIONS, PARTITION_KEY_VALS, PARTITION_PARAMS and other tables. If there are too many partitions, the larger the amount of data in these tables will be. The default heap memory of hiveserver2 is only 256m, so heap is insufficient.

If the number of mapred.reduce.tasks configured by hive-site.xml is large (the default is-1, that is, the number of slave), it will result in more map processes per query job and more partitions, increasing the amount of partition data loaded by a single mapred. The low memory usage of the configuration in mapred-site.xml will also lead to an error during query execution, which can be adjusted appropriately: mapred.child.java.opts=-Xmx512m-XX:+UseConcMarkSweepGC

Solution: partition according to other rules, reduce the number of target table partitions, modify hive-env.sh, and add configuration: export HADOOP_HEAPSIZE=2048

Launch hiveserver2,beeline login to execute any query after hive0.12 upgrade to 0.13. An error is reported:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.unset (Ljava/lang/String;) V

Reason: there is no Configuration.unset (String) method in hadoop1.0.3, compared with API, we can know: Configuration2.2.0, Configuration1.0.4.

Reference: NoSuchMethodError exception when using HIVE 0.13 with Hadoop 1.0.4

Fix: choose one of the following three methods

Add parameter when hiveserver2 starts: hiveserver2-- hiveconf fs.permissions.umask-mode=022

Modify 1.0.3 source code: org/apache/hadoop/hive/ql/exec/Utilities.java, change line 3417 to: conf.set ("fs.permissions.umask-mode", "")

Replace the class into the hive-exec-0.13.0.jar package after recompilation.

Modify hive-site.xml to add the following configuration:

Fs.permissions.umask-mode 022 Setting a value for fs.permissions.umask-mode to work around issue in HIVE-6962. It has no impact in hadoop 1.x line on hdfs operations.

Any SQL after hive0.13.1 upgrade to 0.14 reports an error:

NoSuchMethodError: org.apache.hadoop.mapred.JobConf.unset (Ljava/lang/String;) V

Reason: as above, an incompatible version of hadoop,JobConf inherits Configuration, so there is no unset method.

Reference: https://cwiki.apache.org/confluence/display/Hive/FilterPushdownDev

Explanation: Filter Pushdown, filter chain is used to further improve the data hit rate.

Fix: modify the pushFilters method on line 429 of org/apache/hadoop/hive/ql/io/HiveInputFormat.java

Compile and repackage the first two lines of code using unset after comments:

Javac-cp.: / hive/lib/*:/hadoop/lib/*:/hadoop/* HiveInputFormat.javamkdir-p org/apache/hadoop/hive/ql/io/mv HiveInputFormat*.class org/apache/hadoop/hive/ql/io/jar uf / hive/lib/hive-exec-1.1.0.jar org/apache/hadoop/hive/ql/io/HiveInputFormat*.class

Modified version download address: hive-exec-0.14.0.jar

HQL reports an error when executing a dynamic partitioning statement:

Org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error: Unable to deserialize reduce input key from x1x128x0x0x19x1x255 with properties {columns=reducesinkkey0,reducesinkkey1,reducesinkkey2,reducesinkkey3, serialization.lib=org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe, serialization.sort.order=+, columns.types=int,int,int,int,string,bigint}

Reason: hive0.13.1 bug, if the hive.optimize.sort.dynamic.partition parameter is set (default is true, see hive-default.xml.template), all fields will be sorted during dynamic partitioning, and the fields in group by will also be affected.

Reference: Dynamic sort optimization propagates additional columns even in the absence of order by

Fix: hive0.13.1 can be upgraded to 0.14.

Hive stored procedure (not supported at this time)

Reference: https://issues.apache.org/jira/browse/HIVE-3087

Explanation: this estimate will not be available in the future, because it can be replaced by shell.

Hive1.2 and Hadoop2.6 execute hive client or beeline error:

Exception in thread "main" java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected

Reason: the jline version in YARN is too low.

Solution: replace the jline in hive-lib with the yarn-lib directory:

Cp $HIVE_HOME/lib/jline-2.12.jar $HADOOP_HOME/share/hadoop/yarn/lib/rm-f $HADOOP_HOME/share/hadoop/yarn/lib/jline-0.9.94.jar is all the content of the article "what are the frequently asked questions in Hive?" Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.