Shulou(Shulou.com)05/31 Report--

This article mainly introduces how to integrate sparksql and hive. It is very detailed and has a certain reference value. Friends who are interested must finish it!

Hive configuration

Edit $HIVE_HOME/conf/hive-site.xml and add the following:

Hive.metastore.uris thrift://master:9083 Thrift uri for the remote metastore. Used by metastore client to connect to remote metastore.12345 starts hive metastore starts metastore:$ hive-- service metastore & View metastore:$ jobs [1] + Running hive-- service metastore & close metastore:$kill% 1kill% jobid 1 represents job id1234567891011spark configuration $HIVE_HOME/conf/hive-site.xml copy or soft chain to $SPARK_HOME/conf/ $HIVE_HOME/lib/mysql-connector-java-5.1.12.jar copy or soft chain to $SPARK_HOME/lib/copy or soft chain $SPARK_HOME/lib/ is convenient for spark standalone mode to start spark-sql using 123

Standalone mode

. / bin/spark-sql-- master spark:master:7077-- jars / home/stark_summer/spark/spark-1.4/spark-1.4.1/lib/mysql-connector-java-5.1.12.jar

one

Yarn-client mode

$. / bin/spark-sql-- master yarn-client-- jars / home/stark_summer/spark/spark-1.4/spark-1.4.1/lib/mysql-connector-java-5.1.12.jar execute sql:select count (*) from o2o_app Results: 302Time taken: 0.828 seconds, Fetched 1 row (s) 2015-09-14 18 INFO 2743 158 INFO [main] CliDriver (SessionState.java:printInfo)-Time taken: 0.828 seconds Fetched 1 row (s) spark-sql > 2015-09-14 18 Para43160 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo (59))-Finished stage: org.apache.spark.scheduler.StageInfo@5939ed302015-09-14 18Participation43161 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo (59))-task runtime: (count: 1, mean: 242.000000, stdev: 0.000000, max: 242.000000 Min: 242.000000) 2015-09-14 18 SparkListenerBus 2715 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo (59))-0% 5% 10% 25% 50% 75% 90% 95% 100% 2015-09-14 18 18 Visa 43161 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo (59))-242.0 ms 242.0 ms 242.0 ms 242.0 ms2015-09-14 18 paste 2743 162 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo (59))-fetch wait time: (count: 1 Mean: 0.000000, stdev: 0.000000, max: 0.000000 Min: 0.000000) 2015-09-14 18 ms 27 SparkListenerBus 43162 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo (59))-0% 5% 10% 25% 50% 75% 90% 100% 2015-09-14 18 18 SparkListenerBus 2715 24 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo (59))-0.0 ms 0.0 ms 0.0 ms 0. 0 ms 0.0 ms2015-09-14 18 Groupe 2743 163 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo (59))-remote bytes read: (count: 1 Mean: 31.000000, stdev: 0.000000, max: 31.000000 Min: 31.000000) 2015-09-14 18 SparkListenerBus 2715 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo (59))-0% 5% 10% 25% 50% 75% 90% 95% 100% 2015-09-14 18 18 Visa 27 SparkListenerBus 43 163 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo (59))-31.0 B 31.0 B 31.0 B 31. 0 B 31.0 B2015-09-14 18 purl 2715 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo (59))-task result size: (count: 1 Mean: 1228.000000, stdev: 0.000000, max: 1228.000000 Min: 1228.000000) 2015-09-14 18 scheduler.StatsReportListener 27 SparkListenerBus INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo (59))-0% 5% 10% 25% 50% 75% 90% 95% 100% 2015-09-14 18 Visa 27 SparkListenerBus 43 163 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo (59))-1228.0 B 1228.0 B 1228.0 B 1228.0 B2015-09-14 18 purse 2715 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo (59))-executor (non-fetch) time pct: (count: 1 Mean: 69.834711, stdev: 0.000000, max: 69.834711 Min: 69.834711) 2015-09-14 18 scheduler.StatsReportListener 27 SparkListenerBus INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo (59))-0% 5% 10% 25% 50% 75% 90% 95% 100% 2015-09-14 18 18 SparkListenerBus 43164 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo (59))-70% 70% 70% 70% 70% 70% 70% 2015-09-14 1815 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo (59))-fetch wait time pct: (count: 1 Mean: 0.000000, stdev: 0.000000, max: 0.000000 Min: 0.000000) 2015-09-14 18 SparkListenerBus 2715 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo (59))-0% 5% 10% 25% 50% 75% 90% 95% 100% 2015-09-14 18 SparkListenerBus 27 SparkListenerBus 43 165 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo (59))-0% 0% 0 0% 0% 0% 0% 0% 2015-09-14 1815 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo (59))-other time pct: (count: 1 Mean: 30.165289, stdev: 0.000000, max: 30.165289 Min: 30.165289) 2015-09-14 18 SparkListenerBus 27V 43166 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo (59))-0% 5% 10% 25% 50% 75% 90% 95% 100% 2015-09-14 18 SparkListenerBus 27V 43166 INFO [SparkListenerBus] scheduler.StatsReportListener (Logging.scala:logInfo (59))-30% 30% 30% 30% 30% 30% 30% 3456789101112131415171819202122325262728293031

Yarn-cluster mode

. / bin/spark-sql-- master yarn-cluster-- jars / home/dp/spark/spark-1.4/spark-1.4.1/lib/mysql-connector-java-5.1.12.jarError: Cluster deploy mode is not applicable to Spark SQL shell.Run with-- help for usage help or--verbose for debug output2015-09-14 18 jars 28291 INFO [Thread-0] util.Utils (Logging.scala:logInfo (59))-123456 launch spark-shell not supported by Shutdown hook calledCluster deploy mode

Standalone mode

. / bin/spark-shell-- master spark:master:7077-- jars / home/stark_summer/spark/spark-1.4/spark-1.4.1/lib/mysql-connector-java-5.1.12.jar1

Yarn-client mode

. / bin/spark-shell-- master yarn-client-- jars / home/dp/spark/spark-1.4/spark-1.4.1/lib/mysql-connector-java-5.1.12.jarsqlContext.sql ("from o2o_app SELECT count (appkey,name1,name2)". Collect (). Foreach (println) 1234 is all the content of this article "how to integrate sparksql and hive". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.