Shulou(Shulou.com)05/31 Report--

This article mainly introduces how to configure Spark History Server in a stand-alone environment. The article is very detailed and has certain reference value. Interested friends must read it!

Spark History Server Configuration in a Standalone Environment

First modify two profiles

vim spark-env.shexport SPARK_MASTER_IP=***.***.***.*** export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=18080 -Dspark.history.retainedApplications=10 -Dspark.history.fs.logDirectory=file:///home/zzx/spark-1.1.0-bin-hadoop1/historylogs"export SPARK_CONF_DIR=/home/zzx/spark-1.1.0-bin-hadoop1/confvim spark-defaults.confspark.eventLog.enabled truespark.eventLog.dir file:///home/zzx/spark-1.1.0-bin-hadoop1/historylogsspark.history.fs.logDirectory file:///home/zzx/spark-1.1.0-bin-hadoop1/historylogs

After configuration, remember to source the file path here was originally HDFS path, but in the later test process, no matter how many times the program was run, the History Server page was always blank, and then changed to the local text path.

Start Spark and Spark History Server.

sbin/start-all.shsbin/start-history-server.sh

JPS View Status

Run SparkPi in CLI command-line interface test

bin/run-example SparkPi

After running the program several times, you can access the page according to SPARK_MASTER_IP plus port number 18080

When eclipse runs the program, remember to add a piece of code, because eclipse submission program does not look at the conf file of spark like CLI, so you need to manually add it here. Note that the program finally remembers to close sc, as follows:

val conf = new SparkConf().setAppName("SparkWordCount") .setMaster("local") .set("spark.eventLog.enabled","true") .set("spark.eventLog.dir","/home/zzx/spark-1.1.0-bin-hadoop1/historylogs")val sc = new SparkContext(conf) //Set scsc.stop //Close sc

History information can also be viewed under the configured file path:

The above is all the content of this article "How to configure Spark History Server in a stand-alone environment." Thank you for reading! Hope to share the content to help everyone, more relevant knowledge, welcome to pay attention to the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.