Shulou(Shulou.com)06/02 Report--

This article mainly explains "how to use Spark Stream". The content in the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn how to use Spark Stream.

Pom.xml

Org.apache.spark

Spark-streaming_2.11

2.1.0

Import java.io.OutputStream

Import java.net.ServerSocket

Import java.net.Socket

Import java.util.Arrays

Import java.util.UUID

Import org.apache.spark.SparkConf

Import org.apache.spark.streaming.Durations

Import org.apache.spark.streaming.api.java.JavaDStream

Import org.apache.spark.streaming.api.java.JavaPairDStream

Import org.apache.spark.streaming.api.java.JavaStreamingContext

Import scala.Tuple2

Public class SparkStreamTest {

Public static void main (String [] args) throws Exception {

StartSockerServer (9999)

SparkConf conf = new SparkConf () .setMaster ("local [2]") .setAppName ("NetworkWordCount")

JavaStreamingContext jssc = new JavaStreamingContext (conf, Durations.seconds (1))

JavaDStream lines = jssc.socketTextStream ("localhost", 9999)

JavaDStream words = lines.flatMap (x-> Arrays.asList (x.split (")) .iterator ())

JavaPairDStream pairs = words.mapToPair (s-> new Tuple2 (s, 1)

JavaPairDStream counts = pairs.reduceByKey ((x, y)-> x + y)

Counts.print ()

Jssc.start ()

Jssc.awaitTermination ()

Jssc.close ()

}

Private static void startSockerServer (int port) throws Exception {

New Thread () {

Public void run () {

Try {

ServerSocket server = new ServerSocket (port)

Socket socket = server.accept ()

OutputStream os = socket.getOutputStream ()

For (int I = 0; I < 1000; iTunes +) {

String txt = UUID.randomUUID (). ToString (). ReplaceAll ("-", ")

Txt = "Hello test\ n"

Os.write (txt.getBytes ())

Thread.sleep (10)

}

Socket.close ()

Server.close ()

} catch (Exception e) {

E.printStackTrace ()

}

}

}. Start ()

}

}

Will continue to print out similar results:

Time: 1490947248000 ms

(Hello,99)

(test,99)

Thank you for your reading, the above is the content of "how to use Spark Stream", after the study of this article, I believe you have a deeper understanding of how to use Spark Stream, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.