Network Security Internet Technology Development Database Servers Mobile Phone Android Software Apple Software Computer Software News IT Information

In addition to Weibo, there is also WeChat

Please pay attention

WeChat public account

Shulou

Programming steps of sparkStreaming

2025-01-19 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Internet Technology >

Share

Shulou(Shulou.com)06/03 Report--

(1) StreamingContext

   is similar to spark core programming, when writing SparkStreaming programs, it also needs a general programming entrance-StreamingContext.

Creation of StreamingContext:

Object StreamingContextTest {def main (args: Array [String]): Unit = {val sparkConf = new SparkConf (). SetAppName ("SCTest"). SetMaster ("local [4]") val streamingContext = new StreamingContext (sparkConf, Seconds (2))}}

Note:

If the-- master is specified using local and only one thread is specified when calculating, then only the receiver thread works, and the computing thread does not work, so specify at least 2 threads when specifying the number of threads.

(2) create InputDStream through input source:

After building the StreamingContext, we first need to read the data from the data source for real-time processing:

   InputDStreams refers to the input data stream received from the source of the data stream. Each InputDStream is associated with a Receiver object that receives data from the data source and stores it in memory for later Spark processing.

   Spark Streaming provides two natively supported stream data sources and custom data sources:

   -created directly from StreamingContext API, such as file systems (local and distributed file systems), Socket connections, and Actor for Akka.

   -Kafka, Flume, Kinesis, Twitter, etc., need to rely on external utility classes and external dependencies at run time

   -Spark Streaming also supports user-defined data sources, which require user-defined receiver

Note:

  -master URL cannot use "local" or "local [1]" when running Spark Streaming locally, because when Input DStream is associated with Receiver (such as sockets, Kafka, Flume, etc.), Receiver itself needs a thread to run, and there is no thread to process the received data. Therefore, when running the SparkStreaming program locally, use "local [n]" as the number of master URL,n greater than receiver.

  -when running Spark Streaming on a cluster, the number of CPU cores assigned to the Spark Streaming program must also be greater than the number of receiver, otherwise the system will only accept data and cannot process it.

(3) perform transformation and output operations on DStream, which constitute the logic of late streaming computing. (4) start the process of receiving and processing data through the streamingContext.start () method. (5) use the streamingContext.awaitTermination () method to wait for the program to end (manually stop or error stop). (6) call the streamingContext.stop () method to end the program.

Note when writing sparkStreaming:

After   -streamingContext starts, adding new operations will not work. Be sure to define the logic before starting, that is, after calling the start method, logical operations on the sparkStreaming program are not allowed.

  -StreamingContext is that after the singleton object is stopped, it cannot be restarted unless the task is restarted and the calculation is reperformed.

  -StreamingContext that cannot have two active states in a single jvm for a period of time

  -when calling the stop method of StreamingContext, SparkContext will also be dropped by stop by default. If you want StreamingContext to be closed and you can keep SparkContext, you need to pass the parameter stop SparkContext=false in the stop method.

  -one SparkContext can be used to create multiple StreamingContext, as long as the previous StreamingContext has been stopped.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.

Views: 0

*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.

Share To

Internet Technology

Wechat

© 2024 shulou.com SLNews company. All rights reserved.

12
Report