Shulou(Shulou.com)06/03 Report--

In the operation and maintenance of hadoop, we often encounter some performance problems. On the other hand, performance problems cannot be simply analyzed by web pages and log. Many indicators are needed. And Ganglia is one of the more practical monitoring tools.

Deploy Ganglia, Baidu, many people have shared a lot. Combined with everyone's experience. And add the problems encountered in the installation process, sort out this article.

1. Two machines are prepared.

Server192.168.0.11 (gmetad,web,gmond-master) Client192.168.0.12 (gmond)

2. Software packages that need to be installed on the Server side

Installation of ● epel packages: yum install-y epel-release (resolving the problem that some installation packages cannot be installed in yum)

Installation of ● gmetad: yum install-y ganglia-gmetad ganglia-devel

Installation of ● gmond: yum install-y ganglia-gmond ganglia-gmond-python

Installation of ● rrdtool: yum install-y rrdtool rrdtool-devel

Installation of ● httpd server: yum install-y httpd

● ganglia-web and php installation: yum install-y ganglia-web php

Installation of ● other dependent packages: yum install-y apr-devel zlib-devel libconfuse-devel expat-devel pcre-devel

3. The software package to be installed by the monitored node

Installation of ● epel packages: yum install-y epel-release (resolving the problem that some installation packages cannot be installed in yum)

Installation of ● gmond: yum install-y ganglia-gmond ganglia-gmond-python

4. Installation directory description

● ganglia configuration file directory: / etc/ganglia

● rrd database storage directory: / var/lib/ganglia/rrds

● httpd home site directory: / var/www/html

● ganglia-web installation directory: / usr/share/ganglia

● ganglia-web configuration directory: / etc/httpd/conf.d/ganglia.conf

5. Close SELINUX

Vi / etc/selinux/config changing SELINUX=enforcing to SELINUX=disable; requires restarting the machine.

6. Turn off the firewall

# chkconfig iptables off# chkconfig iptables-listiptables 0:off 1:off 2:off 3:off 4:off 5:off 6:off

7. Configuration / etc/ganglia/gmetad.conf

Modify data_source to:

Data_source "testcluster" 192.168.0.11 gmetad 8650 # destination gmond address and port of data collection (tcp_accept_channel)

8. Configure gmond

/ etc/ganglia/gmond.conf, modify the following (this gmond node is a collection node, this node can be multiple, and finally you need to configure it on gmetad.conf):

Cluster {name = "testcluster" # set the name of the cluster # owner = "unspecified" latlong = "unspecified" url = "unspecified"} # address and port sent to the destination gmond (unicast) udp_send_channel {host=192.168.0.11port = 8649ttl = 1} # receive udp udp_recv_channel {port = 8649} # gmetad if the port tcp_accept_channel {port = 8650gzip_output = no} comes to collect the data request

9. Configure web

Soft connection mode

> ln-s / usr/share/ganglia / var/www/ganglia

You can also copy the contents of / usr/share/ganglia directly to / var/www/ganglia

10. Modify / etc/httpd/conf.d/ganglia.conf to:

Alias / ganglia / usr/share/gangliaOrder deny,allowAllow from all

11. Start the service

# service gmetad start#service gmond start#service httpd restart

At this point, the server side of Ganglia has been deployed.

Configure the client:

twelve。 The client only needs to configure gmond (you need to install yum-y install ganglia-gmond ganglia-gmond-python first)

/ etc/ganglia/gmond.conf, modify the following (this gmond node is a collection node, this node can be multiple, and finally you need to configure it on gmetad.conf):

Cluster {name = "testcluster" # set the name of the cluster # owner = "unspecified" latlong = "unspecified" url = "unspecified"} # address and port sent to the destination gmond (unicast) udp_send_channel {host=192.168.248.130port = 8649ttl = 1} # receive udp udp_recv_channel {port = 8649} # gmetad if the port tcp_accept_channel {port = 8650gzip_output = no} comes to collect the data request

13. Configure HDFS, YARN integrated Ganglia

Modify hadoop-metrics2.properties

# for Ganglia 3.1 support*.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31*.sink.ganglia.period=10# default for supportsparse is false*.sink.ganglia.supportsparse=true*.sink.ganglia.slope=jvm.metrics.gcCount=zero,jvm.metrics.memHeapUsedM=both*.sink.ganglia.dmax=jvm.metrics.threadsBlocked=70 Jvm.metrics.memHeapUsedM=40namenode.sink.ganglia.servers=192.168.0.11:8649 # host Please refer to the definition of datanode.sink.ganglia.servers=192.168.0.11:8649resourcemanager.sink.ganglia.servers=192.168.0.11:8649nodemanager.sink.ganglia.servers=192.168.0.11:8649mrappmaster.sink.ganglia.servers=192.168.0.11:8649jobhistoryserver.sink.ganglia.servers=192.168.0.11:8649 in gmond.conf and note the following parameters If you do not come to the information collection of container, it may result in too much data, and the disk space of ganglia will quickly fill up. # Switch off container metrics*.source.filter.class=org.apache.hadoop.metrics2.filter.GlobFilternodemanager.*.source.filter.exclude=*ContainerResource*

14. HBase integrated Ganglia

Modify hadoop-metrics2-hbase.properties

* .sink.file*.class=org.apache.hadoop.metrics2.sink.FileSink# default sampling period*.period=10*.source.filter.class=org.apache.hadoop.metrics2.filter.GlobFilter*.record.filter.class=$ {* .source.filter.class} * .metric.filter.class = ${* .source.filter.class} hbase.sink.ganglia.record.filter.exclude=*Regions*hbase.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31hbase.sink.ganglia.tagsForPrefix.jvm=ProcessName*.sink.ganglia .period = 20hbase.sink.ganglia.servers=192.168.0.11:8649 # host Please refer to the definition in gmond.conf

15. Copy the configuration file to every machine that needs to be monitored

Copy hadoop-metrics2.properties to the $HADOOP_HOME/etc/hadoop/ directory

Copy hadoop-metrics2-hbase.properties to the $HBASE_HOME/conf directory

Restart the hadoop&hbase software to make it work.

16. Start the gmond on the monitoring side

Service gmond start

Summary of questions:

The client has already passed the information, and can see the overall CPU load and other information.

two。 But the information of each node is empty, showing "no matching metrics detected or rrds not readable"

3. View RRDs information

# cd / var/lib/ganglia/rrds

# ll

Drwxr-xr-x 5 ganglia ganglia 4096 Jan 17 08:50 azcluster

Drwxr-xr-x 2 ganglia ganglia 36864 Jan 17 10:59 _ _ SummaryInfo__

4. The folder name is lowercase

# ll

Drwxr-xr-x 2 ganglia ganglia 32768 Jan 17 10:58 azcbetadnl05.envazure.com

Drwxr-xr-x 2 ganglia ganglia 4096 Jan 17 08:47 azcbetaldapl01.envazure.com

Drwxr-xr-x 2 ganglia ganglia 36864 Jan 17 10:58 _ _ SummaryInfo__

5. The data has been passed on.

# ls azcbetadnl05.envazure.com/ | more

Boottime.rrd

Bytes_in.rrd

Bytes_out.rrd

Cpu_aidle.rrd

Disk_free_absolute_data1.rrd

Disk_free_absolute_data2.rrd

Disk_free_absolute_data3.rrd

Disk_free_absolute_data4.rrd

Disk_free_absolute_data5.rrd

Disk_free_absolute_dev_shm.rrd

Disk_free_absolute_mnt_resource.rrd

.

6. Reason: the folder corresponding to each node in / var/lib/ganglia/rrds is lowercase, so if the node's hostname contains uppercase letters, you will find that the data cannot be found.

Solution: modify gmetad.conf and set the value of case_sensitive_hostnames to 1

# ls / etc/ganglia/

Drwxr-xr-x 2 root root 4096 Jan 17 08:36 conf.d

-rw-r--r-- 1 root root 171Oct 12 2015 conf.php

-rw-r--r-- 1 root root 9834 Jan 17 08:44 gmetad.conf

-rw-r--r-- 1 root root 8756 Jan 17 08:45 gmond.conf

# vi gmetad.conf

# In earlier versions of gmetad, hostnames were handled in a case

# sensitive manner

# If your hostname directories have been renamed to lower case

# set this option to 0 to disable backward compatibility.

# From version 3.2, backwards compatibility will be disabled by default.

# default: 1 (for gmetad

= 3.2)

If case_sensitive_hostnames 1 # is set to 1, uppercase will not be changed to lowercase

7. After the modification, go to the RRDs directory to view the results.

# cd / var/lib/ganglia/rrds/azcluster

Nothing has changed.

# ls-al

Drwxr-xr-x 2 ganglia ganglia 32768 Jan 17 10:58 azcbetadnl05.envazure.com

Drwxr-xr-x 2 ganglia ganglia 4096 Jan 17 08:47 azcbetaldapl01.envazure.com

Drwxr-xr-x 2 ganglia ganglia 36864 Jan 17 10:58 _ _ SummaryInfo__

8. Restart gmetad for the configuration to take effect

# service gmetad restartShutting down GANGLIA gmetad: [OK] Starting GANGLIA gmetad: [OK]

9. You can see that a folder with an uppercase host name has been created

# ls-aldrwxr-xr-x 2 ganglia ganglia 32768 Jan 18 02:09 azcbetadnl05.envazure.comdrwxr-xr-x 2 ganglia ganglia 4096 Jan 18 02:10 AZcbetadnL05.envazure.com

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.