Shulou(Shulou.com)06/03 Report--

Test ideas:

First of all, use the program of the previous article to send network data

Secondly, run the spark program to observe the effect.

Description:

1. The update function is used here

two。 Use checkpoints to ensure status.

SparkStreamingimport org.apache.log4j. {LoggerLevel} import org.apache.spark.streaming. {SecondsStreamingContext} import org.apache.spark. {SparkContextSparkConf} import org.apache.spark.streaming.StreamingContext._object StatefulWordCount {def main (args:Array []) {Logger. () .setLevel (Level.) Logger. () setLevel (Level.) UpdateFunc = (values: [] state:Option []) = > {currentCount = values.foldLeft () (_ + _) previousCount = state.getOrElse () (currentCount + previousCount)} conf = SparkConf (). SetAppName () .setMaster () sc = SparkContext (conf) ssc = StreamingContext (sc () ssc.checkpoint () lines = ssc.socketTextStream (args () args () toInt) words = lines.flatMap (_ .split) () wordCounts = words.map (x = > (x)) stateDstream = wordCounts.updateStateByKey [] (updateFunc) stateDstream.print () ssc.start () ssc.awaitTermination ()}}

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.