Shulou(Shulou.com)06/01 Report--

This article mainly explains the "summary of log files in Hadoop". The content of the explanation in the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "the summary of log files in Hadoop".

Reference: hadoop log location

1. Hadoop MapReduce log classification 1), service log 2), job log

2. The service log of MapReduce in Hadoop version 1.x includes? 1), JobTracker log

JobTracker: on the JobTracker installation node, the default location is

${hadoop.log.dir} / logs/*-jobtracker-*.log, this file is generated every day, the old log suffix is the date, and the log file suffix for the day is ".log", where the default value of ${hadoop.log.dir} is the hadoop installation directory, that is, ${log}.

2), each TaskTracker log

TaskTracker: on each TaskTracker installation node, the default location is

$HADOOP_HOME/logs/*-tasktracker-*.log, this file is generated every day, and the old log is followed by a log. The log file for the day is suffixed with ".log".

3. What are the two parts of the job log?

The job log includes jobhistory log and task log.

4. What is the function of jobhistory log?

Jobhistory log is a job running log, including job start time, end time, start time of each task, end time, all kinds of counter information and so on. Users can parse all kinds of information of job running from this log, which is very valuable information.

5. Where is the jobhistory log stored by default?

The default location is under the ${hadoop.log.dir} / history directory of the node where the JobTracker is located.

6. What options can be used to configure the location of jobhistory logs?

It can be configured through the parameter hadoop.job.history.location.

7. Where is the task log stored?

Each task log is stored on the task running node in the ${hadoop.log.dir} / userlogs// directory.

8. How many sub-files does each task log contain?

Each task contains three log files, stdout, stderr, and syslog.

1), stdout

Stdout: a log printed from standard output.

For example, System.out.println, note that logs printed through standard output in the program are not directly displayed on the terminal, but are saved in this file.

2), syslog

Syslog is a log printed through log4j, which usually contains the most useful information and is the most critical reference log for error debugging.

Thank you for your reading, the above is the content of "summary of log files in Hadoop". After the study of this article, I believe you have a deeper understanding of the summary of log files in Hadoop, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.