Shulou(Shulou.com)06/01 Report--

This article mainly introduces "how to view job history log in spark". In daily operation, I believe many people have doubts about how to view job history log in spark. Xiaobian consulted all kinds of information and sorted out simple and easy operation methods. I hope to help you answer the doubts of "how to view job history log in spark"! Next, please follow the small series to learn together!

SPARK_HOME/conf

spark-defaults.conf Add the following

spark.eventLog.enabled true

spark.eventLog.dir hdfs://master:8020/var/log/spark

spark.eventLog.compress true

spark-env.sh Add the following content

export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=18080 -Dspark.history.retainedApplications=3 -Dspark.history.fs.logDirectory=hdfs:/master.et2:8020/var/log/spark"

Start start-history-server.sh

SPARK_HOME/conf: Execute./ start-history-server.sh

spark job history web: master:18080

This way, after the spark task is completed, you can view the log through the web page

history server-related configuration parameter description

spark.history.updateInterval

Default: 10

The interval, in seconds, between updating log-related information

spark.history.retainedApplications

Default: 50

The number of Application history records saved in memory. If this value is exceeded, the old application information will be deleted. When the deleted application information is accessed again, the page needs to be reconstructed.

spark.history.ui.port

Default: 18080

HistoryServer web port

spark.history.kerberos.enabled

Default: false

Whether to log in to HistoryServer using keranthus is useful for persistent layer located on HDFS of security cluster. If set to true, configure the following two properties

spark.history.kerberos.principal

Default: kerTOS principal name for HistoryServer

spark.history.kerberos.keytab

kersei keytab file location for HistoryServer

spark.history.ui.acls.enable

Default: false

Check acl when authorized users view application information. If enabled, only application owners and users specified by spark.ui.view.acls can view application information; otherwise, no checks are made

spark.eventLog.enabled

Default: false

Whether to log Spark events for refactoring webUI after application completion

spark.eventLog.dir

Default: file://tmp/spark-events

The path to save log-related information can be HDFS path starting with hdfs://or local path starting with file://, all of which need to be created in advance

spark.eventLog.compress

Default: false

Whether to compress Spark events, if spark.eventLog.enabled is true, the default is snappy

SPARK_HISTORY_OPTS in spark-env.sh for those starting with spark.history and spark-defaults.conf for those starting with spark.eventLog

At this point, the study of "how to view job history log in spark" is over, hoping to solve everyone's doubts. Theory and practice can better match to help everyone learn, go and try it! If you want to continue learning more relevant knowledge, please continue to pay attention to the website, Xiaobian will continue to strive to bring more practical articles for everyone!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.