Shulou(Shulou.com)05/31 Report--

This article mainly explains "how to deploy the hadoop environment". The content in the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn how to deploy the hadoop environment.

Preparatory work

The following steps are performed on all nodes

1.1 modify hostname

Vi / etc/sysconfig/network

1.2 turn off SELinux

View SELinux status getenforce

If SELinux is not closed, close as follows

Vi / etc/selinux/config

Modify the SELinux=disabled. Reboot takes effect. You can restart the host after all the settings are completed.

1.3 turn off the firewall

Service iptables stop

Chkconfig iptables off

Chkconfig iptables-list

1.4 Network configuration

Vim / etc/sysconfig/network-scripts/ifcfg-eth0

1.5 modify host

127.0.0.1 localhost# must be configured

# CDH Cluster

192.168.88.11 hadoop1

192.168.88.12 hadoop2

192.168.88.13 hadoop3

1.6 configure hadoop1 to hadoop2 secret-free login

1.7 all nodes are configured with NTP services

All hosts in the cluster must keep their time synchronized, which will cause all kinds of problems if the time difference is large. The specific ideas are as follows:

Master nodes act as ntp servers to synchronize time with the outside world, and then provide time synchronization services to all datanode nodes. All datanode nodes synchronize time based on master nodes.

All nodes install related components: yum install ntp.

After the configuration is completed, boot: chkconfig ntpd on

Check whether the setting is successful: chkconfig-- list ntpd where 2-5 indicates success if the on status is 2-5.

Master node configuration

Before configuration, use ntpdate to manually synchronize the time, lest the time gap between the local machine and the timing center is too big, so that the ntpd can not be synchronized normally. Here we choose 65.55.56.206 as the timing center, ntpdate-u 202.112.10.36

Vi / etc/ntp.conf

# For more information about this file, see the man pages

# ntp.conf (5), ntp_acc (5), ntp_auth (5), ntp_clock (5), ntp_misc (5), ntp_mon (5).

Driftfile / var/lib/ntp/drift

# Permit time synchronization with our time source, but do not

# permit the source to query or modify the service on this system.

Restrict default kod nomodify notrap nopeer noquery

Restrict-6 default kod nomodify notrap nopeer noquery

# Permit all access over the loopback interface. This could

# be tightened as well, but to do so would effect some of

# the administrative functions.

Restrict 127.0.0.1

Restrict-6:: 1

# Hosts on local network are less restricted.

# allow synchronization time of other machines in the private network

Restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

# Use public servers from the pool.ntp.org project.

# Please consider joining the pool (http://www.pool.ntp.org/join.html).

# the most active time server in China: http://www.pool.ntp.org/zone/cn

Server 210.72.145.44 perfer # China National time receiving Center

Server 202.112.10.36 # 1.cn.pool.ntp.org

Server 59.124.196.83 # 0.asia.pool.ntp.org

# broadcast 192.168.1.255 autokey # broadcast server

# broadcastclient # broadcastclient

# broadcast 224.0.1.1 autokey # multicast server

# multicastclient 224.0.1.1 # multicastclient

# manycastserver 239.255.254.254 # manycastserver

# manycastclient 239.255.254.254 autokey # manycastclient

# allow update time by the upper server

# allow upper-layer time server to actively modify local time

Restrict 210.72.145.44 nomodify notrap noquery

Restrict 202.112.10.36 nomodify notrap noquery

Restrict 59.124.196.83 nomodify notrap noquery

# Undisciplined Local Clock. This is a fake driver intended for backup

# and when no outside source of synchronized time is available.

# when an external time server is not available, use local time as the time service

Server 127.127.1.0 # local clock

Fudge 127.127.1.0 stratum 10

# Enable public key cryptography.

# crypto

Includefile / etc/ntp/crypto/pw

# Key file containing the keys and key identifiers used when operating

# with symmetric key cryptography.

Keys / etc/ntp/keys

# Specify the key identifiers which are trusted.

# trustedkey 4 8 42

# Specify the key identifier to use with the ntpdc utility.

# requestkey 8

# Specify the key identifier to use with the ntpq utility.

# controlkey 8

# Enable writing of statistics records.

# statistics clockstats cryptostats loopstats peerstats

Service ntpd start

Ntpstat

It usually takes 5-10 minutes to successfully connect and synchronize.

[root@hadoop1 ~] # netstat-tlunp | grep ntp

Udp 0 0 192.168.88.11 123 0.0.0.0 * 17339/ntpd # #

Udp 0 0127.0.0.1purl 123 0.0.0.0purl * 17339/ntpd

Udp 0 0 0.0.0. 0. 0. 0. 0. 0. 0. 0. 0. 12. 0. 0. 0. 0. 0

Udp 0 0 fe80::20c:29ff:fe7c:123: * 17339/ntpd

Udp 0 0:: 1 123: * 17339/ntpd

Udp 0 0: 123: * 17339/ntpd

[root@hadoop1] # ntpq-p

Remote refid st t when poll reach delay offset jitter

=

202.118.1.130. INIT. 16 u-64 0 0.000 0.000 0.000

# ntpstat

Unsynchronised

Time server re-starting

Polling server every 64 s

After connecting and synchronizing:

Synchronised to NTP server (202.112.10.36) at stratum 3

Time correct to within 275 ms

Polling server every 256 s

# yum install ntp

# chkconfig ntp on

# vim / etc/ntp.conf

Driftfile / var/lib/ntp/drift

Restrict 127.0.0.1

Restrict-6:: 1

# configure the time server as a local time server

Server 192.168.1.135

Restrict 192.168.1.135 nomodify notrap noquery

Server 127.127.1.0 # local clock

Fudge 127.127.1.0 stratum 10

Includefile / etc/ntp/crypto/pw

Keys / etc/ntp/keys

[root@hadoop2 soft] # ntpdate-u hadoop1

2.cloudra installs all nodes

2.1 download cloudera-manager.repo wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo

1 copy the cloudera-manager.repo file to the / etc/yum.repos.d/ folder of all nodes

Mv cloudera-manager.repo / etc/yum.repos.d/

Vi / etc/yum.conf

Timeout=50000

Yum list | grep cloudera

If the version listed is not the one you installed, execute the following command to try again

Yum clean all

Yum list | grep cloudera

Download CDH copy the 3 files of Parcel previously downloaded to the / opt/cloudera/parcel-repo directory (if you do not have this directory, please create it yourself)

Wget http://archive-primary.cloudera.com/cdh6/parcels/5.2.1/CDH-5.2.1-1.cdh6.2.1.p0.12-el5.parcel

The suffix of the wget http://archive-primary.cloudera.com/cdh6/parcels/5.2.1/CDH-5.2.1-1.cdh6.2.1.p0.12-el5.parcel.sha1###.sha1 file is changed to .sha, while leaving only the hash code portion of the content

Wget http://archive-primary.cloudera.com/cdh6/parcels/5.2.1/manifest.json

2.4 install daemons, server, agent on the master [Hadoop1] node (install daemons first)

Wget http://archive-primary.cloudera.com/cm5/redhat/5/x86_64/cm/5.2.1/RPMS/x86_64/cloudera-manager-daemons-5.2.1-1.cm521.p0.109.el5.x86_64.rpm

Wget http://archive-primary.cloudera.com/cm5/redhat/5/x86_64/cm/5.2.1/RPMS/x86_64/cloudera-manager-server-5.2.1-1.cm521.p0.109.el5.x86_64.rpm

Wget http://archive-primary.cloudera.com/cm5/redhat/5/x86_64/cm/5.2.1/RPMS/x86_64/cloudera-manager-agent-5.2.1-1.cm521.p0.109.el5.x86_64.rpm

Yum-- nogpgcheck localinstall cloudera-manager-daemons-5.2.1-1.cm521.p0.109.el6.x86_64.rpm

Yum-- nogpgcheck localinstall cloudera-manager-server-5.2.1-1.cm521.p0.109.el6.x86_64.rpm

Yum-- nogpgcheck localinstall cloudera-manager-agent-5.2.1-1.cm521.p0.109.el6.x86_64.rpm (Note: agent installation requires networking)

2.5 install daemons and agent on slave-1 [hadoop2] and slave-2 [hadoop3] nodes (daemons first)

Wget http://archive-primary.cloudera.com/cm5/redhat/5/x86_64/cm/5.2.1/RPMS/x86_64/cloudera-manager-daemons-5.2.1-1.cm521.p0.109.el5.x86_64.rpm

Wget http://archive-primary.cloudera.com/cm5/redhat/5/x86_64/cm/5.2.1/RPMS/x86_64/cloudera-manager-agent-5.2.1-1.cm521.p0.109.el5.x86_64.rpm

Yum-- nogpgcheck localinstall cloudera-manager-daemons-5.2.1-1.cm521.p0.109.el6.x86_64.rpm

Yum-- nogpgcheck localinstall cloudera-manager-agent-5.2.1-1.cm521.p0.109.el6.x86_64.rpm (Note: agent installation requires networking)

Install JDK and oraclejdk on master, slave-1 and slave-2 nodes

Rpm-ivh jdk-6u31-linux-amd64.rpm

3. Install the mysql database on the master node and configure the database options required by cdh

Yum install mysql-server mysql mysql-devel

Chkconfig mysqld on

Service mysqld start

Mysql-u root

Use mysql

Update user set password=password ('1234') where user='root'

Update user set password=password ('1234') where host='localhost'

Update user set password=password ('1234') where host='hadoop1'

Service mysqld restart

Mysql-u root-p1234

Create database cloudera

4. Configure the cloudera manager database on the master node and start the server and agent programs of cm

1. Copy mysql-connector-java-5.1.7-bin.jar to / usr/share/java and rename mysql-connector-java.jar

two。 Run / usr/share/cmf/schema/scm_prepare_database.sh-h hadoop1 mysql cloudera root 1234

3. Start cm server: service cloudera-scm-server start

4. Add cm server service: chkconfig cloudera-scm-server on

5. Start cm agent: chkconfig cloudera-scm-agent on

6. Add cm agent service: service cloudera-scm-server start

5. Modify the agent configuration files of all nodes

/ etc/cloudera-scm-agent/config.ini changes host in configuration file to cdh-master

6. Configure cloudera manager agent program in slave node

1. Start cm agent: chkconfig cloudera-scm-agent on

two。 Add cm agent service: service cloudera-scm-agent start

7. Test whether agent and server communicate successfully

Service cloudera-scm-server status

Service cloudera-scm-agent status

Netstat-anp | grep 7182

# Port 7182 is opened on the server side for communication with agent

You can view the log when startup fails

Server log / var/log/cloudera-scm-server

Agent log / var/log/cloudera-scm-agent

8 set parcel [master]

Mv CDH-5.2.1-1.cdh6.2.1.p0.12-el5.parcel / opt/cloudera/parcel-repo

[root@hadoop1 parcel-repo] # tail-5 manifest.json

"replaces": "IMPALA, SOLR, SPARK"

"hash": "7dcb31e557a7da951bfb6337e02b0b884aa3d2a2\ n"

}

]

[root@hadoop1 parcel-repo] # tail-1 CDH-5.2.1-1.cdh6.2.1.p0.12-el5.parcel.sha1

7dcb31e557a7da951bfb6337e02b0b884aa3d2a2\ n

[root@hadoop1 parcel-repo] # mv CDH-5.2.1-1.cdh6.2.1.p0.12-el5.parcel.sha1 CDH-5.2.1-1.cdh6.2.1.p0.12-el5.parcel.sha

9. [root@hadoop1 soft] # rpm-ivh oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm all nodes

CDH cluster installation

After the CM installation is successful, the browser enters http://ip:7180,ip as the host ip or hostname installed by CM. The following interface is displayed, and the user name and password are entered into admin to enter the web management interface.

Free version-> continue-> find and select the machine on which you want to install CDH, and click "continue" 192.168.88. [11-13]->

II. Uninstall steps

Record the uninstall process and problems. The CDH environment of the existing environment Cloudera Manager + (1 + 2).

1. First remove all services on the Manage management side.

2. Delete Manager Server

Run on the Manager node

/ usr/share/cmf/uninstall-cloudera-manager.sh if you do not have this script, you can delete it manually and stop the service first:

Service cloudera-scm-server stop

Service cloudera-scm-server-db stop then delete:

Yum remove cloudera-manager-serversudo

Yum remove cloudera-manager-server-db3, delete the CDH service on all CDH nodes, and stop the service first:

Service cloudera-scm-agent hard_stop uninstalls the installed software:

Yum remove 'cloudera-manager-*' hadoop hue-common' bigtop-*'4, delete residual data:

Rm-Rf / usr/share/cmf / var/lib/cloudera* / var/cache/yum/cloudera*

5. Kill drops all Manager and Hadoop processes (optional, this step is not required if you stop Cloud Manager and all services correctly)

$for u in hdfs mapred cloudera-scm hbase hue zookeeper oozie hive impala flume; do sudo kill $(ps-u $u-o pid=); done6, delete the lock file of Manager

Run on the Manager node:

Rm / tmp/.scm_prepare_node.lock at this point, deletion is complete.

/ var/log/cloudera-manager-installer/3.install-cloudera-manager-server.log

Http://archive-primary.cloudera.com/cm5/redhat/5/x86_64/cm/5.2.1/RPMS/x86_64/

Acquire lock unloading installation

Couldn't resolve host 'archive.cloudera.com' "dns8.8.8.8

Note that the hostname should be the same as host. If it is inconsistent, delete it and search again.

Searching for re-uninstall before installing

[root@h02 soft] # service cloudera-scm-agent status

Cloudera-scm-agent dead but pid file exists

[root@client ~] # cd / var/run

[root@client] # rm-f cloudera-scm-agent.pid

An error message was found in the log:

ERROR ENGINE Error in HTTP server: shutting down Traceback (most recent call last)

IOError: [Errno 2] No such file or directory:'/ var/lib/cloudera-scm-agent/uuid'

[root@h02 cloudera-scm-agent] # mkdir / var/lib/cloudera-scm-agent/

[root@h02 cloudera-scm-agent] # chmod 777 / var/lib/cloudera-scm-agent/

Thank you for reading, the above is the content of "how to deploy the hadoop environment". After the study of this article, I believe you have a deeper understanding of how to deploy the hadoop environment, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.