Shulou(Shulou.com)05/31 Report--

This article is to share with you what the logging configuration is like in Spark. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

When testing spark computing, submit the job to yarn (mode-master yarn-cluster). If you want to view the print to the console, this is imposible, because the job is submitted to the yarn cluster, and it is very troublesome for so to read the log on the yarn cluster, but there is a special need to see the print information for debugging or other purposes.

Under the conf directory of Spark, change log4j.properties.template to log4j.properties, and the original content is as follows:

# Set everything to be logged to the console log4j.rootCategory=INFO, console log4j.appender.console=org.apache.log4j.ConsoleAppender log4j.appender.console.target=System.err log4j.appender.console.layout=org.apache.log4j.PatternLayout log4j.appender.console.layout.ConversionPattern=%d {yy/MM/dd HH:mm:ss}% p% c {1}:% m% n

# Settings to quiet third party logs that are too verbose log4j.logger.org.spark-project.jetty=WARN log4j.logger.org.spark-project.jetty.util.component.AbstractLifeCycle=ERROR log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO

Change log4j.rootCategory=INFO and console to log4j.rootCategory=WARN, and console can restrain Spark from typing INFO-level logs to the console. If you want to display comprehensive information, change INFO to DEBUG.

If you want to print the println in the code to the console on the one hand and keep the log output by spark itself on the other hand, you can output it to a log file

Log4j.rootCategory=INFO, console,FILE log4j.appender.console=org.apache.log4j.ConsoleAppender log4j.appender.console.target=System.err log4j.appender.console.layout=org.apache.log4j.PatternLayout log4j.appender.console.layout.ConversionPattern=%d {yy/MM/dd HH:mm:ss}% p% c {1}:% m% n

Settings to quiet third party logs that are too verbose

Log4j.logger.org.eclipse.jetty=WARN log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO

Log4j.appender.FILE=org.apache.log4j.DailyRollingFileAppender log4j.appender.FILE.Threshold=DEBUG log4j.appender.FILE.file=/home/hadoop/spark.log log4j.appender.FILE.DatePattern='.'yyyy-MM-dd log4j.appender.FILE.layout=org.apache.log4j.PatternLayout log4j.appender.FILE.layout.ConversionPattern= [%-5p] [% d {yyyy-MM-dd HH:mm:ss}] [% C {1}:% MVA% L]% m% n

Spark

Log4j.logger.org.apache.spark=INFO

For the above operation, the log of spark is printed to the console on the one hand and written to / home/hadoop/spark.log on the other. This is the inherited feature of the log. It will be improved later. At present, change log4j.rootCategory=INFO and console,FILE to log4j.rootCategory=INFO and FILE.

Thank you for reading! This is the end of this article on "what is the log configuration in Spark?". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.