Shulou(Shulou.com)06/01 Report--

This article introduces the relevant knowledge of "what is the role of Worker and Master in Spark". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Worker and Master are classes used in Spark stand-alone clusters. If you deploy in a yarn environment, you don't need these two classes.

Master is the controller of the Spark independent cluster, Worker is the worker, and a Spark independent cluster needs to start a Master and multiple Worker. Spark provides Master election function to ensure that when Master hangs up, it can select another Master and do a switch action. This principle is similar to ZooKeeper. You can know the concept here.

The first function of Master is to manage the registration and cancellation of Worker, status changes, etc., and maintain the metadata of the cluster; but to manage the submission and operation of Driver.

When the spark program runs driver, it will schedule the execution of the partition task through StandaloneScheduleBackend. At this time, there will be a StandaloneAppClient in the StandaloneScheduleBackend responsible for communicating with Master, sending a RegisterApplication message to Master,Master to record the app information waiting to run, waiting for the scheduling of the schedule method. The RegisterApplication contains the information of the Driver to be executed (Driver is what the main method of our spark program does).

When Master schedules (the schedule method), and then selects a Worker as the executor of the Driver, it sends a LaunchDriver message to Worker,Worker and launches the Driver locally after receiving it.

It should be noted that Master and Worker are machine-level concepts, and Executor and Driver are two different things. ScheduleBackend handles message communication between Driver and Executor.

CoarseGrainedSchedulerBackend is the SchedulerBackend of Driver and is the public part. StandaloneSchedulerBackend in a separate cluster actually inherits CoarseGrainedSchedulerBackend and shares common code that interacts with Executor, such as LaunchTask message processing.

Running on Executor, CoarseGrainedExecutorBackend,CoarseGrainedExecutorBackend is responsible for receiving messages sent by Driver, replying to Driver, etc.

CoarseGrainedSchedulerBackend sends a LaunchTask message to CoarseGrainedExecutorBackend,CoarseGrainedExecutorBackend of Executor. After receiving LaunchTask, call the launchTask method of Executor to execute Task.

This is the end of the content of "what is the role of Worker and Master in Spark". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.