Shulou(Shulou.com)06/03 Report--

The last blog post has introduced the construction of scala development environment, and now get to the point. How to develop our first spark program.

Download the spark installation package and download the address http://spark.apache.org/downloads.html (because the development environment needs to reference spark's jar package)

I downloaded spark-2.1.0-bin-hadoop2.6.tgz because my scalaIDE version is scala-SDK-4.5.0-vfinal-2.11-win32.win32.x86_64.zip

It is best to match the IDE version and the spark version, otherwise, when developing the program, the jar package referenced by the package may be wrong, and so on.

New a scala project, then new a scala object

The code is as follows:

Package com.testimport org.apache.spark.SparkContextimport org.apache.spark.SparkConf / * count the number of character occurrences * / object WordCount {def main (args: Array [String]) {if (args.length)

< 1) { System.err.println("Usage: ") System.exit(1) } val conf = new SparkConf();//创建SparkConf对象 conf.setAppName("Wow,My First Spark Programe");//设置应用程序的名称，在程序运行的监 conf.setMaster("local")//此时，程序在本地运行，不需要安装Spark集群 val sc = new SparkContext(conf);//创建SparkContext对象，通过传入SparkConf实例// val lines = sc.textFile(args(0)); val lines = sc.textFile(args(0)); val words = lines.flatMap{line =>

Line.split ("")}; / / A pair of strings of each line are split into words val pairs = words.map {word = > (word,1)}; val wordCounts = pairs.reduceByKey (_ + _); / / A pair of the same Key accumulates Value (including both Local and Reducer levels Reduce) / / val wordCounts = pairs.reduce ((xMagney) = > (x.)) WordCounts.foreach (wordNumberPair = > println (wordNumberPair._1 + ":" + wordNumberPair)); / / line.flatMap (_ .split (")). Map ((_, 1)). ReduceByKey (_ + _). Collect (). Foreach (println) sc.stop ()}}

3. Configure JDK1.8 and introduce spark jar packages

The jar package of spark is the jar package in the jars directory after the spark installation package has been decompressed. You can import all the jar packages under this directory into the project.

4.run Configurations, input parameters for configuration and operation

Because this code reads a text, in the hadoop file system, you can pass this parameter when you can spark-submit, and in the windows local development environment, you can configure this parameter in eclipse, as shown in the following figure

Val lines = sc.textFile (args (0))

4. Input parameter text and running result

Input parameter text:

Results of run scala Application operation

Ok, the development environment has been built.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.