In addition to Weibo, there is also WeChat
Please pay attention
WeChat public account
Shulou
2025-01-25 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Internet Technology >
Share
Shulou(Shulou.com)06/02 Report--
What are the two ways of submitting tasks in spark02--Standalone mode? I believe many inexperienced people are at a loss about this. Therefore, this paper summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.
I. Standalone-client 's way of submitting tasks
Submit command
. / spark-submit-- master spark://xxx:7077-- class xxx.jar or. / spark-submit-- master spark://xxx:7077-- deploy-mode client-- class xxx.jar
Execution process
1. After the task is submitted in client mode, the Driver process is started on the client side.
2. Driver will apply to Master for resources to launch Application startup.
3. If the resource application is successful, the Driver sends the task to the worker for execution.
4. Worker returns the result of task execution to the driver side.
Summary
The client mode is suitable for testing debuggers. The Driver process is started on the client side, which is the current node that submits the application. You can see the execution of task on the driver side. The client mode cannot be used in the production environment because: if you want to submit 100 application to the cluster to run, and the Driver will be started on the client side every time, it will cause the client Nic traffic to surge for 100 times.
Cluster
Submit command
. / spark-submit-- master spark://xxx:7077-- deploy-mode cluster-- class xxx.jar
Execution process
1. After the application is submitted in cluster mode, it requests Master to start Driver.
2. Master accepts the request and randomly starts the Driver process on a node in the cluster.
3. Apply for resources for the current application after Driver starts.
4. The driver sends task to the worker node for execution.
5. Worker returns the execution and the execution result to the driver.
Summary
The Driver process is started on a Worker in the cluster, and the client cannot view the execution of the task. Suppose you want to submit 100application to the cluster to run, and each Driver will be randomly started on one of the Worker in the cluster, then the problem of the sudden increase in Nic traffic will be scattered on the cluster.
Summarize the two ways of submitting tasks in Standalone. The communication between Driver and cluster includes:
1. Driver is responsible for the application of application resources.
2. Distribution of tasks
3. Recovery of results
4. Monitor the implementation of task
2. Two ways of submitting tasks in Yarn mode
Client
Submit command
. / spark-submit-- master yarn--class xxx.jar or. / spark-submit-- master yarn-client-- class xxx.jar or. / spark-submit-- master yarn--deploy-mode client-- class xxx.jar
Execution process
The client submits an Application and starts a Driver process on the client.
When the application starts, it sends a request to RM (ResourceManager) to start the resource of AM (ApplicationMaster).
RM receives the request and randomly selects a NM (NodeManager) to launch AM. The NM here is equivalent to the Worker node in Standalone.
After AM starts, it requests a batch of container resources from RM to start Executor.
RM will find a batch of NM to return to AM to start Executor.
AM sends a command to NM to start Executor.
After Executor starts, it will reverse register to Driver,Driver and send task to Executor, and the execution and result will be returned to the driver side.
Summary
Yarn-client mode is also suitable for testing, because Driver runs locally, Driver will communicate a lot with Executor in yarn cluster, which will increase the traffic of client network card.
The role of ApplicationMaster:
Request resources for the current Application
Send a message to NameNode to start Executor.
Note: ApplicationMaster has the ability to launchExecutor and request resources, but not job scheduling.
Cluster
Submit command
. / spark-submit-master yarn-cluster-class xxx.jar or. / spark-submit-master yarn--deploy-mode cluster-class xxx.jar
Execution process
The client submits the Application application, sends a request to RS (ResourceManager), and requests to start AM (ApplicationMaster).
After receiving the request, RM randomly launches AM (equivalent to Driver side) on a NM (NodeManager).
AM starts, and AM sends a request to RM, requesting a batch of container to start Executor.
RM returns a batch of NM nodes to AM.
AM connects to NM and sends a request to NM to start Executor.
The Executor is reverse registered to the Driver of the node where the AM is located. Driver sends task to Executor.
Summary
Yarn-Cluster is mainly used in the production environment, because Driver runs in a nodeManager in the Yarn cluster, and the Driver where the task is submitted is located on a random machine, which does not cause a surge in the traffic of a certain machine's network card. The disadvantage is that the log cannot be seen after the task is submitted. Logs can only be viewed through yarn.
The role of ApplicationMaster:
Request resources for the current Application
Send a message to NameNode to start Excutor.
Task scheduling.
After reading the above, have you mastered what are the two ways to submit tasks in spark02--Standalone mode? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!
Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.
Views: 0
*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.
Continue with the installation of the previous hadoop.First, install zookooper1. Decompress zookoope
"Every 5-10 years, there's a rare product, a really special, very unusual product that's the most un
© 2024 shulou.com SLNews company. All rights reserved.