Shulou(Shulou.com)06/03 Report--

I. Log classification

1. Process log

The process log is the log of each hadoop daemon, such as ResourceManager, NodeManager, NameNode, DataNode, and so on. If an exception occurs during the start-up or operation of the cluster, you should first check the process log, which is an important tool for system troubleshooting. The default location is ${HADOOP_HOME} / logs. You can change the default location by specifying the HADOOP_MAPRED_LOG_DIR and YARN_LOG_DIR variables in mapred-env.sh and yarn-env.sh, respectively.

The process log is scrolled, and when the log reaches the specified size, it scrolls to the next file. You can set the file size, the number of reserved files, and so on in $HADOOP_HOME/etc/hadoop/log4j.properties.

2. Job log

The job log records the overall running of the job, as well as counter statistics for the job. For example, how many map and reduce are there in the job, when and which node are started, how many files are read and written by each map, how much CPU time is taken, and so on. Job logs play an important role in system monitoring and tuning.

3. Container log

Container logs are logs recorded by each container of hadoop, which contain important information about errors or failures. If log aggregation is not turned on, it is distributed across nodemanager nodes by default. If the log aggregation option is turned on, it will be uniformly placed in a centralized location (for example, on HDFS). The Container log records all kinds of run-time information and errors that occur when the job is running, which is very helpful for debugging and tuning.

2. Movement of the location of the job log

The job log records the overall situation of the job and changes with the phase of the execution of the job.

1. After the job starts, hadoop will place the job log in the ${yarn.app.mapreduce.am.staging-dir} / ${user} / .staging / {$job_ID} directory. Where:

Yarn.app.mapreduce.am.staging-dir is specified in yarn-site.xml, and the default is / tmp/hadoop-yarn/staging

${user} is the user running the job

For example, the grid user initiates a mr job with job_id as job_1483969906296_0002, and when its run does not end, the log of the job is placed in the / tmp/hadoop-yarn/staging/grid/.staging/job_1483969906296_0002 directory.

2. After the job is completed, the job log will be moved to ${mapreduce.jobhistory.intermediate-done-dir} / ${user}, where ${mapreduce.jobhistory.intermediate-done-dir} can be configured in mapred-site.xml. The default value is ${yarn.app.mapreduce.am.staging-dir} / history/done_intermediate.

3. The new address in step 2 is only a temporary transit station, and hadoop will regularly transfer the logs in this directory to the permanent address: ${mapreduce.jobhistory.done-dir} / ${year} / ${month} / ${day} / 000000. 000000 of them should change. What kind of variable is not determined for the time being and will be added later.

III. Other supplements

1. The jhist file in the job log is in json format, which saves the main information of the job.

2. If you need to view container logs, you'd better turn on log aggregation (configure yarn.log-aggregation-enable as true in yarn-site.xml). You can use yarn logs-applicationId to view the complete container logs.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.