Shulou(Shulou.com)05/31 Report--

This article mainly explains "how to build Spark clusters". The explanation in this article is simple and clear, easy to learn and understand. Please follow the ideas of Xiaobian and go deep into it slowly to study and learn "how to build Spark clusters" together.

From the console we can see that there are two DataNodes, at this point we click on "Live Nodes" to view their information:

From the console we saw our two Datanodes SparkWorker1 and SparkWorker2, exactly what we expected!

Step 3: Start Yarn Cluster

Use the jps command to discover that the ResourceManager process is started on the SparkMaster machine:

On SparkWorker1 and SparkWorker2, the NodeManager process is launched:

Visit http://SparkMaster:8088 on SparkMaster to view ResourceManager health status via the Web Console:

Visit http://SparkWorker1: 8042 on SparkMaster to view NodeManager health status on SparkWorker1 via the Web console:

Visit http://SparkWorker2: 8042 on SparkMaster to view NodeManager health status on SparkWorker2 via the Web console:

Next use "mr-jobhistory-daemon.sh" to start JobHistory Server:

After startup, you can see the task execution history information in JobHistory on the Web console through http://SparkMaster:19888:

Thank you for reading, the above is the content of "how to build Spark cluster", after the study of this article, I believe everyone has a deeper understanding of how to build Spark cluster, the specific use situation still needs to be verified by practice. Here is, Xiaobian will push more articles related to knowledge points for everyone, welcome to pay attention!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.