Shulou(Shulou.com)06/01 Report--

This article mainly explains "how to generate a DriverDescription type message in Standalone cluster mode". The content of the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "how to generate a DriverDescription type message in Standalone cluster mode".

First look at the code, this class code is relatively short, the directory is under deploy/rest/.

Private [spark] class RestSubmissionClientApp extends SparkApplication {/ * * Submits a request to run the application and return the response. Visible for testing. * / def run (appResource: String, mainClass: String, appArgs: Array [String], conf: SparkConf, env: Map [String, String] = Map (): SubmitRestProtocolResponse = {val master = conf.getOption ("spark.master"). GetOrElse {throw new IllegalArgumentException ("'spark.master' must be set.")} val sparkProperties = conf.getAll.toMap val client = new RestSubmissionClient (master) val submitRequest = client.constructSubmitRequest (appResource, mainClass AppArgs, sparkProperties, env) client.createSubmission (submitRequest)} override def start (args: Array [String], conf: SparkConf): Unit = {if (args.length < 2) {sys.error ("Usage: RestSubmissionClient [app resource] [main class] [app args*]") sys.exit (1)} val appResource = args (0) val mainClass = args (1) val appArgs = args.slice (2) Args.length) val env = RestSubmissionClient.filterSystemEnvironment (sys.env) run (appResource, mainClass, appArgs, conf, env)}}

Create a client for RestSubmissionClient, and then submit the message to client in the format:

(appResource, mainClass, appArgs, sparkProperties, env)

Client.createSubmission (submitRequest)

What does client.createSubmission command to do? It is to submit the message to the server, and the real processor is the server, the RestSubmissionServer class or its subclasses. For a stand-alone cluster, it is handled by StandaloneRestServer, so we just look at the processing logic of the submit command.

There are two related functions:

Private method buildDriverDescription and override interface method handleSubmit

The previous method is called in handleSubmit, and the most critical code is two lines:

Val driverDescription = buildDriverDescription (submitRequest) val response = masterEndpoint.askSync [DeployMessages.SubmitDriverResponse] (DeployMessages.RequestSubmitDriver (driverDescription))

Generate a message of type DriverDescription, then send a RequestSubmitDriver message to Master and let Master schedule and execute our spark program, which is driver here.

Next, we enter the Master processing process.

Thank you for your reading, the above is the content of "how to generate a DriverDescription type message in Standalone cluster mode". After the study of this article, I believe you have a deeper understanding of how to generate a DriverDescription type message under Standalone cluster mode. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.