Shulou(Shulou.com)06/03 Report--

Go straight to the code.

Let's talk about the test idea:

The / tmp/sparkStream/ directory that the code monitors

First, create the directory mkdir-p / tmp/sparkStream

Then, run the spark program

Finally, add data files to the monitoring directory / tmp/sparkStream/

Observe the running effect of spark program.

SparkStreamingimport org.apache.log4j. {LoggerLevel} import org.apache.spark.SparkConfimport org.apache.spark.streaming. {SecondsStreamingContext} import org.apache.spark.streaming.StreamingContext._object HdfsWordCount {def main (args: Array []) {Logger.getLogger ("org.apache.spark") .setLevel (Level.WARN) Logger.getLogger ("org.apache.eclipse.jetty.server") .setLevel (Level.OFF) sparkConf = SparkConf (). SetAppName (). SetMaster () ssc = StreamingContext (sparkConf ()) lines = ssc.textFileStream () words = lines.flatMap (_ .split () wordCounts = words.map (x = > (x)). ReduceByKey (_ + _) wordCounts.print () ssc.start () ssc.awaitTermination ()}}

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.