Shulou(Shulou.com)05/31 Report--

What this article shares with you is about how to implement inside JobScheduler. The editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

A key statement in the spark stream program is: ssc.start ()

1. Trace the start method that enters StreamingContext. There is a very key statement, scheduler.start (), which is a JobScheduler (spark stream is used for job scheduling).

The entrance to job scheduling!

2. Include the start method of JobScheduler.

Several key points in this approach are:

EventLoop.start () an event loop that responds to events sent by other components (including job startup, completion, and error reporting).

ReceiverTracker.start () controls the generation of the whole receiver and the acceptance of data.

JobGenerator.start () really starts to generate job

In this method, an event handling circulator eventLoop is also maintained to handle various events.

One of the most critical events is GenerateJobs (time), which is the event that generates the job!

Trace is counted in generateJobs (time)

JobScheduler.receiverTracker.allocateBlocksToBatch (time) distributes the received data Blocks for the current bath.

Graph.generateJobs (time): generates the corresponding job based on the output action of the currently written program and encapsulates it into the collection.

Finally passed

Submit the job to executor

Go back and look at startFirstTime () in jobGenerator.start ().

Private def startFirstTime () {val startTime = new Time (timer.getStartTime ()) graph.start (startTime-graph.batchDuration) timer.start (startTime.milliseconds) logInfo ("Started JobGenerator at" + startTime)}

The first startup will start a timer, which will continuously give the message loop body in the jobGenerator based on the duration bath!

The message loop body in jobGenerator will constantly remove messages for processing.

The above is how to carry out JobScheduler insider implementation, the editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.