Shulou(Shulou.com)06/01 Report--

This article introduces the problems encountered by SparkSQL in visiting Hive and what are the solutions. The content is very detailed. Interested friends can use it for reference. I hope it will be helpful to you.

You need to copy the hive-site.xml of hadoop's core-site.xml,hive to project first.

Test code

Error

View the source code

Solution method

Add spark-hive_2.11-2.4.2.jar and spark-hive-thriftserver_2.11-2.4.2.jar under $HIVE_HOME/lib to project

Continue to report errors

View the source code

Enter ConfVars

It is found that the variable defined in ConfVars does not have METASTORE_CLIENT_SOCKET_LIFETIME, while HiveConf.java comes from hive-exec-1.1.0-cdh6.7.0.jar, which proves that there is no if parameter in hive1.1.0.

Solution method

Change hive dependency to 1.2.1

Continue to report errors

Solution method

This is due to the fact that the remote end does not start hive, and metastore needs to be configured when starting hive.

On the SparkSQL access to Hive problems and what is the solution to share here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.