Shulou(Shulou.com)05/31 Report--

This article introduces the relevant knowledge of "what is the dynamic Job generation method". In the operation process of actual cases, many people will encounter such difficulties. Next, let Xiaobian lead you to learn how to deal with these situations! I hope you can read carefully and learn something!

A key statement in the spark stream program is ssc.start()

1, trace into the StreamingContext start method, there is a very critical statement scheduler.start(), is a JobScheduler(spark stream used for job scheduling)

Entry for job scheduling!

2, included in the JobScheduler start method.

A few key points in this approach are:

eventLoop.start() An event circulator that responds to events sent by other components (including job starts, completions, and error reports).

receiverTracker.start() controls the generation of the entire receiver, and the reception of data

jobGenerator.start() really starts job generation

An event-handling circulator eventLoop is also maintained in this method to handle various events

One of the most critical events is GenerateJobs(time), which is the event of generating jobs!!

Track included generateJobs(time)

jobScheduler.receiverTracker.allocateBlocksToBatch(time) Distributes received blocks for the current bath.

graph.generateJobs(time): Generate jobs based on the output actions of the currently written program and encapsulate them into collections.

finally passed

Submit job to executor

Go back and look at startFirstTime() in jobGenerator.start()

private def startFirstTime() { val startTime = new Time(timer.getStartTime()) graph.start(startTime - graph.batchDuration) timer.start(startTime.milliseconds) logInfo("Started JobGenerator at " + startTime)}

The first start starts a timer that will continue to loop messages to jobGenerator based on the duration bath!

The message loop body in jobGenerator will continuously remove messages for processing

"Job dynamic generation method is what" content is introduced here, thank you for reading. If you want to know more about industry-related knowledge, you can pay attention to the website. Xiaobian will output more high-quality practical articles for everyone!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.