Shulou(Shulou.com)05/31 Report--

This article focuses on "what the Web monitoring page of spark is like". Friends who are interested may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn what the Web monitoring page of spark is like.

You can see the initialization UI code in SparkContext:

/ / Initialize the Spark UIprivate [spark] val ui: Option [SparkUI] = if (conf.getBoolean ("spark.ui.enabled", true)) {Some (SparkUI.createLiveUI (this, conf, listenerBus, jobProgressListener, env.securityManager,appName))} else {/ / For tests, do not enable the UI None} / / Bind the UI before starting the task scheduler to communicate// the bound port to the cluster manager properlyui.foreach (_ .bind ()

Create a SparkUI object:

Def createLiveUI (sc: SparkContext, conf: SparkConf, listenerBus: SparkListenerBus, jobProgressListener: JobProgressListener, securityManager: SecurityManager, appName: String): SparkUI = {create (Some (sc), conf, listenerBus, securityManager, appName, jobProgressListener = Some (jobProgressListener))}

Enter the create method:

SparkUI inherits WebUI:

The initialize () implementation method of SparkUI:

2 、 stagesTab:

4 、 EnvironmentTab:

6 、 createStaticHandler:

After executing the initialize () method, we go back to SparkContext's Initialize the spark UI:

As you can see from the above code, the local JettyServer with port number 4040 has been started

The above code analysis:

1. Create ContextHandlerCollection and set handlers to ContextHandlerCollection

2. Add a filter:

3. Create a Jetty Server, bind the port number, create a QueuedThreadPool, and set it to the Server:

5. Try to start startServiceOnPort several times. If the startup fails, change the new port number (rule is 1+oldPort) and restart:

At this point, I believe you have a deeper understanding of "what the Web monitoring page of spark is like". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.