Shulou(Shulou.com)06/01 Report--

This article mainly introduces how to install CDH5.4.1 online, the article is very detailed, has a certain reference value, interested friends must read it!

Download CENTOS6.4

Http://118.122.119.70/CentOS-6.4-x86_64-bin-DVD1.iso?fid=KBiGdobo4XpP2az01*useRyNKDoASHsDAQAAAPMPsesUnrepQ15p8WWIN7Ynh4Ft&mid=666&threshold=150&tid=4990D3C32C91D7AF85B132EF51EE881D&srcid=119&verno=1

Configure Hostnam

Network configuration

Install the Apache server

Turn off firewall and selinux and configure trust relationship

Set NTP time synchronization

Add ordinary user hadoop and set sudo permissions

Installation package preparation

Execution file http://archive.cloudera.com/cm5/installer

Rpm package: http://archive.cloudra.com/cm5/repo-as-tarball/

Give permission to chmod 777 cloudra-manager-installer.bin

[root@localhost ~] # service iptables stop

Iptables: Flushing firewall rules: [OK]

Iptables: Setting chains to policy ACCEPT: filter [OK]

Iptables: Unloading modules:

Root@localhost ~] # chkconfig iptables off

[OK]

[root@localhost ~] # vi / etc/sysconfig/selinux

Modify selinu

# This file controls the state of SELinux on the system.

# SELINUX= can take one of these three values:

# enforcing-SELinux security policy is enforced.

# permissive-SELinux prints warnings instead of enforcing.

# disabled-No SELinux policy is loaded.

SELINUX=disabled

# SELINUXTYPE= can take one of these two values:

# targeted-Targeted processes are protected

# mls-Multi Level Security protection.

SELINUXTYPE=targeted

[root@localhost ~] # reboot

[root@localhost ~] # reboot

[root@localhost ~] # setup

[root@localhost ~] # service network restart

Shutting down interface eth2: [OK]

Shutting down loopback interface: [OK]

Bringing up loopback interface: [OK]

Bringing up interface eth2: [OK]

[root@localhost ~] # ifconfig

Eth2 Link encap:Ethernet HWaddr 00:0C:29:30:FE:A3

Inet addr:192.168.141.11 Bcast:192.168.141.255 Mask:255.255.255.0

Inet6 addr: fe80::20c:29ff:fe30:fea3/64 Scope:Link

Modify IP and hostname mapping relationship vi / etc/hosts

192.168.141.11 CDH01

[root@localhost ~] # hostname CDH01

[root@localhost ~] # hostname

CDH01

[root@localhost ~] # vi / etc/sysconfig/network modify hostname

NETWORKING=yes

HOSTNAME=CDH01

Check to see if java is installed (uninstall it once installed)

[root@CDH01 ~] # rpm-qa | grep jdk

[root@CDH01 ~] # rpm-qa | grep java

Add hadoop user

[root@CDH01 ~] # adduser hadoop

[root@CDH01 ~] # passwd hadoop

Root@CDH01 ~] # su hadoop

[hadoop@CDH01 root] $whoami

Hadoop

Hadoop@CDH01 root] $cd / etc/

[hadoop@CDH01 etc] $ls-l | grep sudo

-rw-r-. 1 root root 1786 Sep 25 2012 sudo.conf

-rMurray, Murray. 1 root root 4002 Mar 1 2012 sudoers

Drwxr-x---. 2 root root 4096 Feb 21 2013 sudoers.d

-rw-r-. 1 root root 3181 Jan 23 2013 sudo-ldap.conf

[hadoop@CDH01 etc] $su

Password:

[root@CDH01 etc] # vi sudoers

Add after root ALL= (ALL) ALL

Hadoop ALL= (root) NOPASSWD:ALL

[root@CDH01 etc] # sudo ls / root/ will not make a mistake.

[root@CDH01 etc] # ssh-keygen-t rsa

[root@CDH01 etc] # ssh-copy-id CDH01

[root@CDH01 etc] # No password is required for ssh CDH01 login

[root@CDH01 ~] # rpm-qa | grep httpd

Httpd-tools-2.2.15-26.el6.centos.x86_64

Httpd-2.2.15-26.el6.centos.x86_64

Root@CDH01 ~] # service httpd status

Httpd is stopped

[root@CDH01 ~] # sudo service httpd start

[root@CDH01 ~] # sudo chkconfig httpd on

[root@CDH01 ~] # reboot

Download the file http://archive.cloudera.com/cm5/installer/5.4.1/cloudera-manager-installer.bin

Download the file http://archive.cloudera.com/cm5/repo-as-tarball/5.4.1/cm5.4.1-centos6.tar.gz (6 stands for centos6.x)

[root@CDH01 /] # cd / opt

[root@CDH01 opt] # pwd

/ opt

[hadoop@CDH01 opt] $sudo rm-rf *

[hadoop@CDH01 opt] $sudo mkdir softwares

[hadoop@CDH01 opt] $ls

Softwares

[hadoop@CDH01 opt] $sudo yum install-y lrzsz

[root@CDH01 softwares] # sudo ls-la / opt

Total 12

Drwxr-xr-x. 3 root root 4096 Dec 18 09:15.

Dr-xr-xr-x. 22 root root 4096 Dec 18 08:06..

Drwxr-xr-x 2 root root 4096 Dec 18 09:25 softwares

[root@CDH01 softwares] # sudo chown-R hadoop:hadoop / opt

[root@CDH01 softwares] # sudo ls-la / opt

Total 12

Drwxr-xr-x. 3 hadoop hadoop 4096 Dec 18 09:15.

Dr-xr-xr-x. 22 root root 4096 Dec 18 08:06..

Drwxr-xr-x 2 hadoop hadoop 4096 Dec 18 09:25 softwares

[root@CDH01 softwares] # tar-zxf cm5.4.1-centos6.tar.gz-C / var/www/html

[root@CDH02 softwares] # cd / var/www/html

[root@CDH01 html] # sudo mkdir-p cm5/redhat/6/x86_64/

[root@CDH01 html] # sudo mv cm cm5/redhat/6/x86_64/

Modify yum source

[root@CDH01 html] # cd / etc/yum.repos.d/

[root@CDH01 yum.repos.d] # ls

CentOS-Base.repo CentOS-Debuginfo.repo CentOS-Media.repo CentOS-Vault.repo

[root@CDH01 yum.repos.d] # sudo vi cloudera-manager.repo

[cloudera-manager]

Name=cloudera-manager

Baseurl= http://CDH01/cm5/redhat/6/x86_64/cm/5/

Enabled=1

Gpgcheck=0

[root@CDH02 yum.repos.d] # yum repolist [generated YumSource]

Loaded plugins: fastestmirror, refresh-packagekit, security

Loading mirror speeds from cached hostfile

Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=os&infra=stock error was

14: PYCURL ERROR 6-"Couldn't resolve host 'mirrorlist.centos.org'"

Cloudera-manager | 951B 00:00

Cloudera-manager/primary | 4.3 kB 00:00

Cloudera-manager 7/7

Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=extras&infra=stock error was

14: PYCURL ERROR 6-"Couldn't resolve host 'mirrorlist.centos.org'"

Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=updates&infra=stock error was

14: PYCURL ERROR 6-"Couldn't resolve host 'mirrorlist.centos.org'"

Repo id repo name status

Base CentOS-6-Base 0

Cloudera-manager cloudera-manager 7

Extras CentOS-6-Extras 0

Updates CentOS-6-Updates 0

Repolist: 7

[root@CDH02 softwares] # wget http://archive.cloudera.com/cm5/installer/5.9.0/cloudera-manager-installer.bin

[root@CDH01 softwares] #. / cloudera-manager-installer.bin [root users and networking]

[root@CDH01 cloudera-manager-installer] # ls

0.check-selinux.log 2.install-oracle-j2sdk1.7.log

1.install-repo-pkg.log 3.install-cloudera-manager-server.log

[root@CDH01 ~] # cd / opt/softwares/

[root@CDH01 softwares] # ls

Cloudera-manager-installer.bin cm5.4.1-centos6.tar.gz

Next- > next- > yes- > next- > yes

Visit 192.168.141.11VR 7180

[root@CDH01 softwares] # cd / var/www/html

[root@CDH01 html] # ls

Cm5

[root@CDH01 html] # mkdir parcels [http://archive.cloudera.com/cdh6/parcels/5.9.0/]

[root@CDH01 html] # ls

Cm5 parcels

[root@CDH01 html] # cd parcels

[root@CDH01 parcels] # ls

CDH-5.4.1-1.cdh6.4.1.p0.6-el6.parcel manifest.json

CDH-5.4.1-1.cdh6.4.1.p0.6-el6.parcel.sha1

[root@CDH01 parcels] # mv CDH-5.4.1-1.cdh6.4.1.p0.6-el6.parcel.sha1 CDH-5.4.1-1.cdh6.4.1.p0.6-el6.parcel.sha

There are three files for accessing http://192.168.141.11/parcels/

Remote Parcel repository URL http://192.168.141.11/parcels/

Select the version of CDH

CDH-5.4.1-1.cdh6.4.1.p0.6

Custom repository

Http://192.168.141.11/cm5/redhat/6/x86_64/cm/5/

Install the Oracle Java SE Development Kit (JDK) tick

Single user ticking

Log in to all hosts as hadoop

Root user

Cloudera recommends that / proc/sys/vm/swappiness be set to 0. Currently set to 60. Use the sysctl command to change the setting at run time and edit / etc/sysctl.conf to save the setting after restart. You can continue with the installation, but you may encounter problems, and Cloudera Manager reports that your host is not performing well due to swapping. The following hosts are affected:

[root@CDH01 parcels] # sudo sysctl vm.swappiness=0

Vm.swappiness = 0

[root@CDH01 parcels] # vi / etc/sysctl.conf

Vm.swappiness=0

Large transparent pages has been enabled, which can cause significant performance problems. Kernel with version "CentOS release 6.4 (Final)" and release "2.6.32-358.el6.x86_64" has set enabled to "[always] never"

And set defrag to "[always] never". Run "echo never > / sys/kernel/mm/redhat_transparent_hugepage/defrag" to disable this setting, and then add the same command to an init script, such as / etc/rc.local, so that it will be set when the system restarts. Alternatively, upgrade to RHEL 6.5 or later, which does not have this error. The following hosts will be affected:

Root@CDH01 parcels] # sudo echo never > / sys/kernel/mm/redhat_transparent_hugepage/defrag

[root@CDH01 parcels] # sudo vi / etc/rc.local

Add echo never > / sys/kernel/mm/redhat_transparent_hugepage/defrag

Add Cloudera Management Service [do not install components yet]

Use embedded database-"continue until complete

Add Service zookeper

[root@CDH01 cloudera-scm-agent] # service cloudera-scm-agent restart [possible permission problem, chmod 777 / var/log/cloudera-scm-*]

Http://192.168.141.11:7180/cmf/login

[root@CDH01 cloudera-scm-server] # netstat-ano | grep 7180

Tcp 00 0.0.0.0 7180 0.0.0.0 * LISTEN off (0.00Uniplex 0)

Tcp 00 192.168.141.11 tcp 7180 192.168.141.1 ESTABLISHED off 51316

Tcp 00 192.168.141.11 ESTABLISHED off 7180 192.168.141.1 ESTABLISHED off

Tcp 00 192.168.141.11 ESTABLISHED off 7180 192.168.141.1 ESTABLISHED off

Tcp 0 0 192.168.141.11 TIME_WAIT timewait 54184 192.168.141.11)

Tcp 0 192.168.141.11 TIME_WAIT timewait 54204 192.168.141.11)

Tcp 00 192.168.141.11 tcp 7180 192.168.141.1 ESTABLISHED off 51312

Tcp 00 192.168.141.11 tcp 7180 192.168.141.1 ESTABLISHED off 51313

Tcp 00 192.168.141.11 tcp 7180 192.168.141.1 51310

[root@CDH02 bin] # cd / opt/cloudera/parcels/CDH/bin

Configuration-> hdfs permissions-"Save changes -" HDFS restart

Root@CDH02 bin] # / usr/java/jdk1.7.0_67-cloudera/bin/jps

18694 EventCatcherService

3614 NameNode

3639 DataNode

18733 Jps

2655 Main

4807 Main

3598 Bootstrap

5149 NodeManager

4828 Main

4762 AlertPublisher

3589 SecondaryNameNode

5372 ResourceManager

4787 Main

3587 QuorumPeerMain

[root@CDH02 CDH] # bin/yarn jar share/doc/hadoop-0.20-mapreduce/examples/hadoop-examples-2.6.0-mr1-cdh6.4.1.jar

Bin/yarn jar share/doc/hadoop-0.20-mapreduce/examples/hadoop-examples-2.6.0-mr1-cdh6.4.1.jar wordcount / a / output01

Configure-"Action -" enable ha

The above is all the contents of the article "how to install CDH5.4.1 online". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.