Shulou(Shulou.com)06/01 Report--

This article mainly introduces the relevant knowledge of "springboot integrates spark and uses spark-sql". The editor shows you the operation process through an actual case. The operation method is simple, fast and practical. I hope this article "springboot integrates spark and uses spark-sql" can help you solve the problem.

First, add related dependencies:

4.0.0 org.springframework.boot spring-boot-starter-parent 1.5.6.RELEASE com.cord spark-example 1.0-SNAPSHOT spark-example http://www.example.com UTF-8 UTF-8 1.8 2.10.3 1.8 1.8 org.springframework.boot spring-boot-starter 1.5.6.RELEASE Org.springframework.boot spring-boot-starter-logging org.apache.spark spark-core_2.10 1.6.1 provided org.slf4j slf4j-log4j12 log4j Log4j org.apache.spark spark-sql_2.10 1.6.1 provided org.apache.spark spark-hive_2.10 1.6.1 provided org.scala-lang scala-library ${scala.version} provided mysql mysql-connector-java 5 . 1.22 org.apache.maven.plugins maven-shade-plugin org.springframework.boot spring-boot-maven-plugin 1.5.6.RELEASE False false *: * META-INF/*.SF META-INF/*.DSA META-INF/*.RSA META-INF/spring.handlers META-INF/spring.factories META-INF/spring.schemas Com.cord.StartApplication package shade

It is important to pay attention to the log modules excluded from the dependency and the special packaging methods.

Define the configuration class:

SparkContextBean.class

@ Configurationpublic class SparkContextBean {private String appName = "sparkExp"; private String master = "local"; @ Bean @ ConditionalOnMissingBean (SparkConf.class) public SparkConf sparkConf () throws Exception {SparkConf conf = new SparkConf () .setAppName (appName) .setMaster (master); return conf;} @ Bean @ ConditionalOnMissingBean public JavaSparkContext javaSparkContext () throws Exception {return new JavaSparkContext (sparkConf ()) @ Bean @ ConditionalOnMissingBean public HiveContext hiveContext () throws Exception {return new HiveContext (javaSparkContext ());}.}

Startup class:

StartApplication.class

@ SpringBootApplicationpublic class StartApplication implements CommandLineRunner {@ Autowired private HiveContext hc; public static void main (String [] args) {SpringApplication.run (StartApplication.class, args);} @ Override public void run (String... Args) throws Exception {DataFrame df = hc.sql ("select count (1) from LCS_DB.STAFF_INFO"); List result = df.javaRDD (). Map ((Function) row-> {return row.getLong (0);}) .collect (); result.stream () .forEach (System.out::println);}

Execution method:

Spark-submit\-- class com.cord.StartApplication\-- executor-memory 4G\-- num-executors 8\-- master yarn-client\ / data/cord/spark-example-1.0-SNAPSHOT.jar 's content on "springboot's method of integrating spark and using spark-sql" ends here. Thank you for reading. If you want to know more about the industry, you can follow the industry information channel. The editor will update different knowledge points for you every day.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.