Shulou(Shulou.com)06/01 Report--

This article mainly explains "the steps of installing hadoop cluster in cloudra". Interested friends may wish to have a look at it. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn the steps of installing a hadoop cluster in cloudra.

The following steps are performed on all nodes

1.1 modify hostname

Vi / etc/sysconfig/network

1.2 turn off SELinux

View SELinux status getenforce

If SELinux is not closed, close as follows

Vi / etc/selinux/config

Modify the SELinux=disabled. Reboot takes effect. You can restart the host after all the settings are completed.

1.3 turn off the firewall

Sudo service iptables stop

Sudo chkconfig iptables off

Sudo chkconfig iptables-list

1.4 Network configuration

Vim / etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE= "eth0"

BOOTPROTO= "static"

IPADDR=192.168.1.110

NM_CONTROLLED= "yes"

ONBOOT= "yes"

TYPE= "Ethernet"

DNS1=8.8.8.8

DNS2=8.8.4.4

GATEWAY=192.168.1.1

1.5 modify host

127.0.0.1 localhost# must be configured

# CDH Cluster

192.168.88.149 h01

192.168.88.148 h02

192.168.88.150 h03

Configure secret login

two。 Download in advance (both CM and CDH versions are 5.2.0 in this tutorial)

2.1 download cloudera-manager.repo download cloudera-manager.repo # 333 http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo

2.2 download CM installer http://archive.cloudera.com/cm5/installer/5.2.0/cloudera-manager-installer.bin

Download CM rpms and download all http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.2.0/RPMS/x86_64/ in the directory

2.4 to download the CDH Parcel file, you only need to download the three files shown in the figure

# http://archive.cloudera.com/cdh6/parcels/5.2.0/CDH-5.2.0-1.cdh6.2.0.p0.36-el6.parcel

# http://archive.cloudera.com/cdh6/parcels/5.2.0/CDH-5.2.0-1.cdh6.2.0.p0.36-el6.parcel.sha1 # .sha1 file suffix is changed to .sha, while leaving only the hash code part of the content

# http://archive.cloudera.com/cdh6/parcels/5.2.0/manifest.json

3.Cloudera Manager installation [all nodes]

3.1 copy the cloudera-manager.repo file to the / etc/yum.repos.d/ folder on all nodes

Mv cloudera-manager.repo / etc/yum.repos.d/

3.2verifying whether the repo file works [root@h01 ~] #

Vi / etc/yum.conf

Timeout=50000

Yum list | grep cloudera

If the version listed is not the one you installed, execute the following command to try again

Yum clean all

Yum list | grep cloudera

Copy the previously downloaded rpms file to all nodes (any directory)

Change to the rpms directory and execute

Yum-y install * .rpm

Download CDH copy the 3 files of the previously downloaded Parcel to the / opt/cloudera/parcel-repo directory (if you do not have this directory, please create it yourself)

Mv manifest.json / opt/cloudera/parcel-repo

Root@h02 soft] # mv CDH-5.2.0-1.cdh6.2.0.p0.36-el6.parcel / opt/cloudera/parcel-repo

Mv CDH-5.2.0-1.cdh6.2.0.p0.36-el6.parcel.sha1 / opt/cloudera/parcel-repo

Mv CDH-5.2.0-1.cdh6.2.0.p0.36-el6.parcel.sha1 CDH-5.2.0-1.cdh6.2.0.p0.36-el6.parcel.sha

So far, there are three files under / opt/cloudera/parcel-repo: CDH-5.2.0-1.cdh6.2.0.p0.36-el6.parcel

Mainfest.json

CDH-5.2.0-1.cdh6.2.0.p0.36-el6.parcel.sha

Install daemons, server, agent (daemons first) (rpm-ivh * .rpm) on the master node

Yum-- nogpgcheck localinstall cloudera-manager-daemons-5.2.1-1.cm521.p0.109.el6.x86_64.rpm

Yum-- nogpgcheck localinstall cloudera-manager-server-5.2.1-1.cm521.p0.109.el6.x86_64.rpm

Yum-- nogpgcheck localinstall cloudera-manager-agent-5.2.1-1.cm521.p0.109.el6.x86_64.rpm (Note: agent installation requires networking)

Install daemons and agent on slave-1 and slave-2 nodes (install daemons first)

Yum-- nogpgcheck localinstall cloudera-manager-daemons-5.2.1-1.cm521.p0.109.el6.x86_64.rpm

Yum-- nogpgcheck localinstall cloudera-manager-agent-5.2.1-1.cm521.p0.109.el6.x86_64.rpm (Note: agent installation requires networking)

Install JDK and oraclejdk on master, slave-1, slave-2 nodes

Rpm-ivh jdk-6u31-linux-amd64.rpm

# Please perform the following steps only on the primary node:

Copy the downloaded bin file to any directory of the main node

Enter the directory and give executable permissions to the bin file

Sudo chmod + x. / cloudera-manager-installer.bin

Install CM

. / cloudera-manager-installer.bin

/ etc/cloudera-scm-server/db.properties

Then all the next and yes will be fine, because the required rpm packages have been downloaded in advance, and the process will proceed very quickly.

Master node configuration cloudera manager

. Start cm server: service cloudera-scm-server start

4. Add cm server service: chkconfig cloudera-scm-server on

5. Start cm agent: chkconfig cloudera-scm-agent on

6. Add cm agent service: service cloudera-scm-server start

IX. Modify the agent configuration files of all nodes

/ etc/cloudera-scm-agent/config.ini changes host in configuration file to cdh-master

Configure the cloudera manager agent program on the slave node

1. Start cm agent: chkconfig cloudera-scm-agent on

two。 Add cm agent service: service cloudera-scm-server start

11. Test whether agent and server communicate successfully

Service cloudera-scm-server status

Service cloudera-scm-agent status

Netstat-anp | grep 7182

# Port 7182 is opened on the server side for communication with agent

You can view the log when startup fails

Server log / var/log/cloudera-scm-server

Agent log / var/log/cloudera-scm-agent

CDH cluster installation

After the CM installation is successful, the browser enters http://ip:7180,ip as the host ip or hostname installed by CM. The following interface is displayed, and the user name and password are entered into admin to enter the web management interface.

II. Uninstall steps

Record the uninstall process and problems. The CDH environment of the existing environment Cloudera Manager + (1 + 2).

1. First remove all services on the Manage management side.

2. Delete Manager Server

Run on the Manager node

/ usr/share/cmf/uninstall-cloudera-manager.sh if you do not have this script, you can delete it manually and stop the service first:

Service cloudera-scm-server stop

Service cloudera-scm-server-db stop then delete:

Yum remove cloudera-manager-serversudo

Yum remove cloudera-manager-server-db3, delete the CDH service on all CDH nodes, and stop the service first:

Service cloudera-scm-agent hard_stop uninstalls the installed software:

Yum remove 'cloudera-manager-*' hadoop hue-common' bigtop-*'4, delete residual data:

Rm-Rf / usr/share/cmf / var/lib/cloudera* / var/cache/yum/cloudera*

5. Kill drops all Manager and Hadoop processes (optional, this step is not required if you stop Cloud Manager and all services correctly)

$for u in hdfs mapred cloudera-scm hbase hue zookeeper oozie hive impala flume; do sudo kill $(ps-u $u-o pid=); done6, delete the lock file of Manager

Run on the Manager node:

Rm / tmp/.scm_prepare_node.lock at this point, deletion is complete.

/ var/log/cloudera-manager-installer/3.install-cloudera-manager-server.log

Http://archive-primary.cloudera.com/cm5/redhat/5/x86_64/cm/5.2.1/RPMS/x86_64/

At this point, I believe you have a deeper understanding of the "steps for cloudra to install hadoop clusters". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.