Shulou(Shulou.com)05/31 Report--

How to run the Spark example program? in view of this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

I. installation and configuration

Download and extract Spark to the specified directory. If you do not install JDK and Scala, you need to download JDK and Scala to the specified directory and set SPKARK_HOME (JAVA_HOME, SCALA_HOME).

Start the Spark or Spark cluster

If you only run Spark on a single node, you don't need to start Spark, just press "three, run the example program" to run. If you need to run the Spark sample program on the cluster, you need to start the Spark cluster first. Add the slave node names (or IP addresses) that have been logged into each other without a password into the conf/slaves file of the master node, and then copy the entire directory of the configured master node's Spark to all other slave nodes. Run the startup cluster script directly:

. / start-all.sh# shuts down the cluster. / stop-all.sh II, run the example program 1. Run on a single node

There is no need to start the cluster to run the example on a single node:

. / bin/run-example SparkPi 10./bin/run-example graphx.LiveJournalPageRank follower.txt-- numEPart=2# runs the Python example program. / bin/spark-submit examples/src/main/python/pi.py 10./bin/spark-submit examples/src/main/python/PageRank.py follower.txt 102. Submit it to the cluster to run

. / bin/spark-submit-- master spark://A281:7077-- class org.apache.spark.examples.graphx.LiveJournalPageRank lib/spark-examples-1.6.0-hadoop2.6.0.jar followers.txt-- numEPart=2

The answer to the question about how to run the Spark example program is shared here. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel to learn more about it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.