Lesson 10: interpretation of Spark Streaming source code

The contents of this issue:

1. Data reception architecture design pattern

2. Thorough research on the source code of data reception.

1. The process of accepting data in Receiver is similar to MVC mode:

The relationship between Receiver,ReceiverSupervisor and Driver is equivalent to Model,Control,View, or MVC.

Model is Receiver, storage data Control, that is, ReceiverSupervisor,Driver is to obtain metadata, that is, View.

2. The location information of the data will be encapsulated in RDD.

3. Receiver accepts the data and gives it to ReceiverSupervisor to store the data.

4. ReceiverTracker is by sending one Job after another. There is only one Task in each Job, and there is only one ReceiverSupervisor in each Task. Use this function to start each Receiver.

Let's simply take a look at the Receiver startup process. The application first starts the start method of receiverTracker through the start method of JobScheduler:

Def start (): Unit = synchronized {if (eventLoop! = null) return / / scheduler has already been startedlogDebug ("Starting JobScheduler") eventLoop = new EventLoop [JobSchedulerEvent] ("JobScheduler") {override protected def onReceive (event: JobSchedulerEvent): Unit = processEvent (event) override protected def onError (e: Throwable): Unit = reportError ("Error in job scheduler", e)} eventLoop.start () / / attach rate controllers of input streams to receive batch completion updatesfor {inputDStream