Shulou(Shulou.com)05/31 Report--

This article shows you how to achieve the first test of Spark Streaming, the content is concise and easy to understand, it will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

First try of Spark Streaming

Yum install nc.x86_64

. / bin/spark-shell-- total-executor-cores 34

Type in the following program:

Import org.apache.spark.SparkConf

Import org.apache.spark.streaming. {Seconds, StreamingContext}

Import org.apache.spark.storage.StorageLevel

Val ssc = new StreamingContext (sc, Seconds (1))

Val lines = ssc.socketTextStream ("hostname", 9999, StorageLevel.MEMORY_AND_DISK_SER)

Val words = lines.flatMap (_ .split (""))

Val wordCounts = words.map (x = > (x, 1)). ReduceByKey (_ + _)

WordCounts.print ()

Ssc.start ()

Ssc.awaitTermination ()

On the hostname machine, after installing nc, nc-lk 9999

Constantly entering characters, such as hello world, will constantly count the character frequency of each input data on the machine that starts spark-shell.

The above content is how to realize the first test of Spark Streaming. Have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.