Shulou(Shulou.com)06/03 Report--

Hadoop concept

Hadoop is a software framework capable of distributed processing of a large amount of data. Hadoop processes data in a reliable, efficient, and scalable manner.

Hadoop is reliable because it assumes that computing elements and storage will fail, so it maintains multiple copies of working data to ensure that processing can be redistributed against failed nodes.

Hadoop is efficient because it works in parallel and speeds up processing through parallel processing.

Hadoop is also scalable and can handle PB-level data.

In addition, Hadoop relies on community services, so it is relatively low-cost and can be used by anyone.

Hadoop is a distributed computing platform that makes it easy for users to structure and use. Users can easily develop and run applications that deal with huge amounts of data on Hadoop. It mainly has the following advantages:

1. High reliability. The ability of Hadoop to store and process data bit by bit is trustworthy.

two。 High scalability. Hadoop distributes data and completes computing tasks among available computer clusters, which can be easily extended to thousands of nodes.

3. High efficiency. Hadoop can move data dynamically between nodes and ensure the dynamic balance of each node, so the processing speed is very fast.

4. High fault tolerance. Hadoop can automatically save multiple copies of data and automatically reassign failed tasks.

5. Low cost. Compared with all-in-one machines, commercial data warehouses and data marts such as QlikView and Yonghong Z-Suite, hadoop is open source, so the software cost of the project will be greatly reduced.

Hadoop indicator type

The metrics of Hadoopde daemons can be divided into different groups according to the context to which they belong, as follows:

JVM metrics: these metrics are generated by JVM running in the cluster, including JVM heap size and garbage collection related metrics, such as current heap memory (MemHeapUsed) usage and total GC count (GcCount).

RPC metrics: metrics in the context of rpc include hostnames and ports, as well as metrics such as number of bytes occurring (SentBytes), current punch connections (NumOpenConnections), and number of authentication failures.

DFS metrics: dfs context includes metrics related to NameNode, HDFS file system, DataNodes, and JournalNodes. DFS metrics can tell whether there are a large number of file creation and deletion operations in the cluster.

Log messages for Hadoop

You can access hadoop log messages for Spark and other jobs by browsing individual log files or through Hadoop's built-in web interface. Most of the time, it is better to access the logs through the web interface because you can save time and quickly find out the cause of performance problems or job failures:

Hadoop generates two main types of logs:

It generates logs for daemons such as NameNode and DataNode. Daemon logs are mainly used by administrators because they help to troubleshoot unexpected failures of key Hadoop services such as DataNode and NameNode.

Hadoop also generates logs for each application running in the cluster, and hadoop application logs can be used by developers to understand the reasons for job failures and performance degradation.

You can view hadoop logs in a variety of ways

Hadoop web UI, especially ResourceManager webUi, can avoid the trouble of accessing log storage location and viewing log files. You can also view logs through JobHistory web UI.

Check the log information directly from the log file

For some application logs, aggregate them to HDFS storage if log aggregation is enabled.

Check through the yarn command:

The yarn application command manages the following tasks:

List the applications running in the cluster

Kill a running application

Gets the status of the running application.

View the Yarn application

Yarn application-list can retrieve a list of all jobs, regardless of their status. Jobs can have the following states: All, NEW, NEW_SAVING, SUMBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, and KILLED. Specify the-appStates option

Yarn application-list-appStates running

Check the status of the application

Yarn application-status

Jobs being executed by kill

Yarn application-kill

Check node status

Yarn node-all-list lists all nodes in the cluster and their status

Get Job Log

The syntax of yarn logs fame and fortune:

Yarn logs-applicationId

You can only get the logs of running jobs that have ended.

Location where Hadoop stores logs

1. Hdfs: this is the location where hadoop creates a staging directory to store job execution files, such as the job.xml that contains the hadoop parameter for running the job.

2. NodeManager local directory: this is the directory created on the local file system where hadoop stores Shell scripts produced by the NodeManager service to execute the ApplicationMaster container. You can use the yarn.nodemanager.local.dir parameter in the yarn-site.xml file to specify the NodeManger local directory location.

This parameter provides a list of directories where NodeManager stores its local files. The local file directory ${yarn.nodemanager.local-dir} / usercache/user/.... of the actual application under these directories. Each NodeManager is in the local application cache under the NodeManager local directory

3. NodeManger log directory: this is the local directory on linux, where NodeManager stores the actual log files of applications that users are running. All containers that execute jobs on this node's NodeManager and their application logs are stored in this directory. Use the yarn.nodemanager.log-dirs parameter to specify the location of the NodeManager log directory.

There is no need to worry that the nm-local-dirs directory will fill up the job files in the appcache subdirectory, which will be automatically deleted when the job is completed. But some jobs do contain large files, the configuration property yarn.nodemanager.delete.debug-delay-sec specifies the time to keep the local log directory after the application, the configuration time expires, and NodeManager's deletionservicehi deletes the application's local file directory structure.

Hadoop storage log aggregation

When log aggregation is enabled, nodemanager connects all container logs to a file and saves it in HDFS, and you can use the yarn.nodemanager.remote-app-log-dir parameter to configure where hadoop stores aggregate logs in hdfs. Generally configured in / tmp/hadoop/logs/

There are three ways to get application logs

1. Obtain from hdfs

2. From web UI via hadoop, click applicationmaster in the unfinished app, and click logs below the tab.

3. View from JobHistoryServer UI after the job is completed.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.