Shulou(Shulou.com)06/03 Report--

Title: introduction to the monitoring data of DKhadoop big data processing platform

The ranking of the top 50 companies of domestic big data in 2018 has been announced, and quickly broke into the top 50 with the appearance of a dark horse and won a number of laurels. Hanlp natural language processing technology also won the "2018 China data Star Technology" award. Those who are interested in this list can look for it. This article continues the previous "DKM platform monitoring parameters description", and continues to introduce and explain the platform monitoring parameters in the fast big data integrated processing architecture.

DKhadoop big data processing platform architecture installation-related articles have been shared, the details can be found to see. In the previous article, we have explained the monitoring parameters such as cluster average load, cluster disk usage, HDFS monitoring interface, Hbase monitoring interface and so on. Today, we will finish the introduction of the remaining monitoring parameters, and the introduction of the monitoring parameters of the fast big data processing platform will be complete.

1. Yarn monitoring interface

(1) execution of failed applications

Monitor the total number of failed applications in yarn resource management

The vertical axis represents the number of applications, per unit

The horizontal axis represents time, in minutes.

(2) submitted applications

Monitor the number of applications that have been submitted in yarn resource management

The vertical axis represents the number of applications, per unit

The horizontal axis represents time, in minutes.

(3) running applications

Monitor the number of running applications in yarn resource management

The vertical axis represents the number of applications, per unit

The horizontal axis represents time, in minutes.

(4) applications waiting to be executed

Monitor the number of applications waiting for execution in yarn resource management

The vertical axis represents the number of applications, per unit

The horizontal axis represents time, in minutes.

(5) completed applications

Monitor the number of completed applications in yarn resource management

The vertical axis represents the number of applications, per unit

The horizontal axis represents time, in minutes.

(6) applications that have been kill

Monitor the number of applications killed in yarn resource management

The vertical axis represents the number of applications, per unit

The horizontal axis represents time, in minutes.

2. Spark monitoring interface

Note: (monitoring data is not available until spark runs the task)

(1) maximum available memory

Monitor the maximum available memory in a Spark cluster

The vertical axis represents the memory capacity, in MB

The horizontal axis represents time, in minutes.

(2) memory used

Monitor the memory already used in the Spark cluster

The vertical axis represents the memory capacity, in MB

The horizontal axis represents time, in minutes.

(3) remaining memory

Monitor the remaining memory in the Spark cluster

Vertical axis represents memory capacity, unit (MB)

The horizontal axis represents time, in minutes.

(4) number of tasks in Spark

Monitor the number of tasks in the Spark cluster

The vertical axis represents the memory capacity, in MB

The horizontal axis represents time, in minutes.

(5) the number of tasks running in the Spark

Monitor the number of tasks running in the Spark

The vertical axis represents the memory capacity, in MB

The horizontal axis represents time, in minutes.

(6) number of stage running

Monitor the number of stage running in the Spark cluster

The vertical axis represents the number of stage, per unit

The horizontal axis represents time, in minutes.

(7) number of failed stage

Monitor the number of failed stage in the Spark cluster

The vertical axis represents the number of stage, per unit

The horizontal axis represents time, in minutes.

(8) number of stage ready

Monitor the number of stage ready in the Spark cluster

The vertical axis represents the number of stage, per unit

The horizontal axis represents time, in minutes.

Note:

(the time of the horizontal axis represents the collection time, usually once every minute)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.