Shulou(Shulou.com)05/31 Report--

This article introduces the relevant knowledge of "what is the role of Yarn Client and Yarn cluster". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

1. Take a look at the architecture diagram of Yarn client.

Yarn Client first starts Driver on the client side, and runs the main function on the Client side. At this time, the jar package in which the main function is located has not yet been submitted to the Yarn cluster. SparkContext starts running and starts YarnClientSchedulerBackend. Then YarnClientSchedulerBackend submits an application to Yarn to execute ApplicationMaster (a class in the spark-yarn project), and the Yarn cluster finds a Container to execute ApplicationMaster. Note: the difference between this and yarn cluster is that only ApplicationMaster is started on Container, not Driver, because it has already been started. Then ApplicationMaster is responsible for resource allocation, dispatches Container to run Executor, and the scheduled Executor communicates with Driver on the client side.

2. Take a look at the architecture diagram of Yarn cluster

Yarn cluster is much simpler. The client does not execute anything. The jar is submitted to the Yarn cluster, and the Yarn cluster finds a Container to start ApplicationMaster. At this time, the difference between ApplicationMaster and yarn client comes. At this time, ApplicationMaster will then start Driver, that is, Driver and ApplicationMaster are in the same place, and then the Executor started by ApplicationMaster also communicates with Driver. Note that the Driver and ApplicationMaster here are in the same place and running in the same process.

Finally, Yarn client is used for debugging environments, while Yarn cluster is used for production environments.

This is the end of the content of "what are the functions of Yarn Client and Yarn cluster". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.