Shulou(Shulou.com)06/02 Report--

This article mainly explains "where the Hadoop log exists". The content in the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "where the Hadoop log exists".

1. Hadoop version 1.x

The service logs of MapReduce in Hadoop 1.x include JobTracker logs and individual TaskTracker logs, and their log locations are as follows:

JobTracker: on the JobTracker installation node, the default location is

${hadoop.log.dir} / logs/*-jobtracker-*.log, this file is generated every day, the old log suffix is the date, and the log file suffix for the day is ".log", where the default value of ${hadoop.log.dir} is the hadoop installation directory, that is, ${log}.

TaskTracker: on each TaskTracker installation node, the default location is

$HADOOP_HOME/logs/*-tasktracker-*.log, this file is generated every day, and the old log is followed by a log. The log file for the day is suffixed with ".log".

The job log includes jobhistory log and task log, in which jobhistory log is the job running log, including job start time, end time, start time, end time, various counter information and so on. Users can parse all kinds of information about job running from this log, which is very valuable information. The default storage location is under the ${hadoop.log.dir} / history directory of the node where the JobTracker resides, which can be configured with the parameter hadoop.job.history.location. Each task log is stored on the task running node in the ${hadoop.log.dir} / userlogs// directory. Each task contains three log files, namely stdout, stderr and syslog. Stdout is the log printed through standard output, such as System.out.println. Note that the log printed by standard output in the program is not directly displayed on the terminal, but is saved in this file. Syslog is a log printed through log4j, which usually contains the most useful information and is the most critical reference log for error debugging.

2. Hadoop 2.x version

The service logs of the YARN system in Hadoop 2.x include ResourceManager logs and individual NodeManager logs, and their log locations are as follows:

The location of the ResourceManager log is the yarn-*-resourcemanager-*.log in the logs directory under the Hadoop installation directory

The location of the NodeManager log is the yarn-*-nodemanager-*.log in the logs directory under the hadoop installation directory on each NodeManager node.

The application log includes the jobhistory log and the Container log, where the jobhistory log is the application running log, including the application start time, end time, start time and end time of each task, various counter information and so on.

Container logs contain ApplicationMaster logs and ordinary Task logs, which are stored in the application_xxx directory in the userlogs directory under the Hadoop installation directory, where the ApplicationMaster log directory name is container_xxx_000001, and the ordinary task log directory name is container_xxx_000002,container_xxx_000003, … ., like Hadoop 1.x, each directory contains three log files: stdout, stderr, and syslog, and the exact meaning is the same.

Thank you for your reading, the above is the content of "where the Hadoop log exists", after the study of this article, I believe you have a deeper understanding of where the Hadoop log exists, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.