Shulou(Shulou.com)05/31 Report--

Today, I will talk to you about how to analyze the nature of Ganglia and the configuration of SparkOnGanglia. Many people may not know much about it. In order to make you understand better, the editor has summarized the following contents for you. I hope you can get something according to this article.

Preface

Ganglia is an open source monitoring project launched by UC Berkeley that is designed to measure thousands of nodes. Each computer runs a daemon named gmond that collects and sends metrics such as processor speed, memory usage, and so on. It will be collected from the operating system and the specified host. The host that receives all the measurement data can display the data and pass a compact form of the data into the hierarchy. It is precisely because of this hierarchical pattern that Ganglia can be well extended. Gmond brings very little system load, which makes it a piece of code running on each computer in the cluster without affecting user performance.

A Ganglia component

The Ganglia monitoring suite consists of three main parts: gmond,gmetad, and the web interface, often referred to as ganglia-web.

Gmond: a daemon that runs on every node that needs to be monitored, collects monitoring statistics, sends and receives statistics on the same multicast or unicast channel. If he is a mute=no, he will collect basic indicators, such as system load (load_one) and CPU utilization. It also sends metrics that users customize by adding C/Python modules. If he is a deaf=no, he aggregates all metrics sent from other hosts and stores them in memory buffers.

Gmetad: also a daemon, it periodically checks the gmonds, pulls data from there, and stores their metrics in the RRD storage engine. He can query multiple clusters and aggregate metrics. It is also used to generate the web front end of the user interface.

Ganglia-web: as the name implies, he should be installed on a machine with gmetad running in order to read the RRD file. A cluster is a logical grouping of host and metric data, such as database servers, web servers, production, testing, QA, etc., which are completely separate and you need to run separate gmond instances for each cluster.

Generally speaking, each node in the cluster needs a receiving gmond, and each website needs a gmetad.

Second, installation dependence

Note: it is recommended to use superuser installation throughout the process. In this example, the installation environment centos6.6

1.#yum install-y gcc gcc-c++ libpng freetype zlib libdbi apr* libxml2-devel pkg-config glib pixman pango pango-devel freetye-devel fontconfig cairo cairo-devel libart_lgpl libart_lgpl-devel pcre* rrdtool*

Install expat dependencies

1. # cd ~

2.#wget http://jaist.dl.sourceforge.net/project/expat/expat/2.1.0/expat-2.1.0.tar.gz

3.#tar-xf expat-2.1.0.tar.gz & & cd expat-2.1.0 &. / configure-- prefix=/usr/local/expat & & make & & make install & & cd.

For 64-bit operating systems, you need to manually copy the dynamic link library to lib64

1.#mkdir / usr/local/expat/lib64 & & cp-a / usr/local/expat/lib/* / usr/local/expat/lib64/

Fourth, install confuse

1.#wget http://ftp.twaren.net/Unix/NonGNU//confuse/confuse-2.7.tar.gz

2.#tar-xf confuse-2.7.tar.gz & & cd confuse-2.7 &. / configure CFLAGS=-fPIC-- disable-nls-- prefix=/usr/local/confuse & & make & & make install & & cd.

64bit machines need to copy dynamic link libraries:

1.#mkdir-p / usr/local/confuse/lib64 & & cp-a-f / usr/local/confuse/lib/* / usr/local/confuse/lib64/

5. Install ganglia

1.#wget http://jaist.dl.sourceforge.net/project/ganglia/ganglia%20monitoring%20core/3.6.0/ganglia-3.6.0.tar.gz

2.#tar-xf ganglia-3.6.0.tar.gz & & cd ganglia-3.6.0 &. / configure-- with-gmetad-- enable-gexec-- with-libconfuse=/usr/local/confuse-- with-libexpat=/usr/local/expat-- prefix=/usr/local/ganglia-- sysconfdir=/etc/ganglia & & make & & make install & & cd.

Sixth, server configuration.

1.#mkdir-p / var/lib/ganglia/rrds & & mkdir-p / var/lib/ganglia/dwoo & & chown-R root:root / var/lib/ganglia

Modify / etc/ganglia/gmetad.conf file

1.data_source "glo" master (master maps the host Ip to be monitored, in this case only one machine, that is, master. If it is not necessary to fill in the port, the default is 8649

The "glo" here represents the name of the cluster, followed by the host information contained in the cluster, that is, the host to be monitored.)

2.setuid_username "root"

Add a self-startup script

1.#cp-f ganglia-3.6.0/gmetad/gmetad.init / etc/init.d/gmetad & & cp-f / usr/local/ganglia/sbin/gmetad / usr/sbin/gmetad & & chkconfig-- add gmetad

Start the gmetad service

1.#service gmetad start

Seeing Starting GANGLIA gmetad: [OK] means it's working properly. Verify that the gmetad is normal through telnet localhost 8651

Seven client configuration (gmond node)

This machine is installed as follows:

1.#cp-f ganglia-3.6.0/gmond/gmond.init / etc/init.d/gmond & & cp-f / usr/local/ganglia/sbin/gmond / usr/sbin/gmond & & chkconfig-- add gmond & & gmond-- default_config > / etc/ganglia/gmond.conf

Appropriate modifications are required for the generated default configuration file / etc/ganglia/gmond.conf

Globals {

User = root / * user running Ganglia * /

Host_dmax = 120 / * secs * /

Send_metadata_interval = 15 / * interval between sending data * /

}

Cluster {

Name = "glo" / * Cluster name * /

Owner = "root" / * user running Ganglia * /

}

Udp_send_channel {

# mcast_join = 239.2.11.71 / * comment out Multicast * /

Host = 192.168.1.108 * send to the machine where gmetad is installed * /

}

Udp_recv_channel {# accepts UDP package configuration

# mcast_join = 239.2.11.71

# bind = 239.2.11.71

}

Where name is the grouping that will be carried out on the server side and is the data source of the server side. Next, start the service.

1.#service gmond start

Seeing Starting GANGLIA gmetad: [OK] indicates that the startup is successful. If there is a failure, you can change the debug in gmond.conf from 0 to 100, read more logs, and then troubleshoot.

8 WEB configuration on the server side

PHP programs rely on Apache to run, so you need to install the following dependencies

1.# yum-y install php httpd

2.# service httpd start / / start the httpd service

Nine install ganglia web

1.#cd / root

2.#wget http://jaist.dl.sourceforge.net/project/ganglia/ganglia-web/3.5.10/ganglia-web-3.5.10.tar.gz

3.#tar-xf ganglia-web-3.5.10.tar.gz & & cd ganglia-web-3.5.10 & & make install & & cd.

This generates the ganglia directory under / var/www/html/

Note:

Ganglia access failed:

There was an error collecting ganglia data (127.0.0.1 Permission denied): fsockopen error:

Resolve:

Selinux:vi / etc/selinux/config needs to be closed, and the machine needs to be restarted to change SELINUX=enforcing to SELINUX=disable;.

You can use the command setenforce 0 to close selinux without restarting, refresh the page and access it. But this method is only a stopgap measure. If you want to permanently modify the selinux settings, you still need to use the first method.

Restart the httpd server to see the effect

1.#service httpd restart

Use http://master/ganglia to view the corresponding ganglia information. (note: master is the hostname of the host running gmetad)

Configuration in SparkOnGanglia (provided that SPARK with ganglia components is compiled)

(this example spark version 1.5.1)

Add at $SPARK_HOME/conf/metrics.properties

* .sink.ganglia.class=org.apache.spark.metrics.sink.GangliaSink

* .sink.ganglia.host=master

* .sink.ganglia.port=8649

* .sink.ganglia.period=2

* .sink.ganglia.unit=seconds

* .sink.ganglia.ttl=1

* .sink.ganglia.mode=unicast

Then view the Metric entry in ganglia web

After reading the above, do you have any further understanding of how to analyze the nature of Ganglia and the configuration of SparkOnGanglia? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.