How to develop Spark2.0 with Eclipse
Editor to share with you how to use Eclipse to develop Spark2.0, I believe that most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to know it!
1. First of all, I used the scala version of ide. This can be downloaded from the official website. Let's first introduce my version: hadoop2.7.2+spark2.0+scala2.11+java1.7.
First open eclipse, set up workspace, and you can start development.
two。 New scala Project. Name a project.
3. Create a new scala object. Start writing code.
Note that the relevant packages need to be imported. There are mainly scala packages and spark packages. The packages needed by spark are mainly under the jar file in the spark installation directory.
4. Cut the crap and go straight to the code.
Object TestSparkStreaming {
Def main (args: Array [String]): Unit = {
Import org.apache.spark._
Import org.apache.spark.streaming._
Import org.apache.spark.streaming.StreamingContext._ / / not necessary since Spark 1.3
/ / Create a local StreamingContext with two working thread and batch interval of 1 second.
/ / The master requires 2 cores to prevent from a starvation scenario.
Val conf = new SparkConf () .setMaster ("local [2]") .setAppName ("NetworkWordCount")
Val ssc = new StreamingContext (conf, Seconds (5))
Val lines = ssc.socketTextStream ("master", 9999)
/ / val lines = ssc.textFileStream ("/ home/usr/temp/")
Val words = lines.flatMap (_ .split (""))
Import org.apache.spark.streaming.StreamingContext._ / / not necessary since Spark 1.3
/ / Count each word in each batch
Val pairs = words.map (word = > (word, 1))
Val wordCounts = pairs.reduceByKey (_ + _)
/ / Print the first ten elements of each RDD generated in this DStream to the console
WordCounts.print ()
Ssc.start () / / Start the computation
Ssc.awaitTermination ()
}
}
5. Connect port 9999 nc-lk 9999 to the linux terminal
Enter some words
6. You can see that the console of eclipse will output the results
The above is all the contents of the article "how to develop Spark2.0 with Eclipse". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!
