Shulou(Shulou.com)06/01 Report--

This article shows you how to use sbt to build a spark project, the content is concise and easy to understand, can definitely brighten your eyes, through the detailed introduction of this article, I hope you can get something.

Using Intellij to build sbt Project scala uses 2.10.4

Name: = "gstorm" version: = "1.0" version: = "2.10.4" val overrideScalaVersion = "2.11.8" val sparkVersion = "2.0.0" val sparkXMLVersion = "0.3.3" val sparkCsvVersion = "1.4.0" val sparkElasticVersion = "2.3.4" val sscKafkaVersion = "2.0.1" val sparkMongoVersion = "1.0.0" val sparkCassandraVersion = "1.6.0" / / Override Scala Version to the above 2.11.8 versionivyScala: = ivyScala.value map {_ .copy (overrideScalaVersion = true)} resolvers + + = Seq ("All Spark Repository-> bintray-spark-packages" at "https://dl.bintray.com/spark-packages/maven/")libraryDependencies + + = Seq (" org.apache.spark "%" spark-core "% sparkVersion exclude (" jline ") ), "org.apache.spark"% "spark-sql"% sparkVersion excludeAll (ExclusionRule (organization = "jline"), ExclusionRule ("name", "2.12")), "org.apache.spark"% "spark-hive"% sparkVersion, "org.apache.spark"% "spark-yarn"% sparkVersion, "com.databricks"% "spark-xml"% sparkXMLVersion "com.databricks"% "spark-csv"% sparkCsvVersion, "org.apache.spark"% "spark-graphx"% sparkVersion, "org.apache.spark"% "spark-catalyst"% sparkVersion, "org.apache.spark"% "spark-streaming"% sparkVersion, / / "com.101tec"% "zkclient"% "0.9" "org.elasticsearch"% "elasticsearch-spark"% sparkElasticVersion, / / "org.apache.spark"% "spark-streaming-kafka-0-102.11"% sscKafkaVersion, "org.mongodb.spark"% "mongo-spark-connector_2.11"% sparkMongoVersion, "com.stratio.datasource"% "spark-mongodb_2.10"% "0.11.1" "dibbhatt"kafka-spark-consumer"1.0.8", "net.liftweb"lift-webkit"2.6.2")

WordCount.scala

Import org.apache.spark.sql.SparkSessionobject WordCount {def main (args: Array [String]): Unit = {val spark = SparkSession .builder () .appName ("Spark SQL Example") .master ("local [2]") .config ("spark.sql.codegen.WordCount") "true") .getOrCreate () val sc = spark.sparkContext val textFile = sc.textFile ("hdfs://hadoop:9000/words.txt") val wordCounts = textFile.flatMap (line = > line.split (")) .map (word = > (word, 1)) .reduceByKey ((a, b) = > a + b) wordCounts.collect.foreach (println)}} the above is how to use sbt to build spark projects Have you learned any knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.