Shulou(Shulou.com)06/03 Report--

1. Installation mode

Offline deployment installation of CDH, namely Parcel package (recommended)

two。 Role planning

Roles corresponding to the three nodes:

3. Basic environment configuration (on each node)

(1) turn off the firewall

# / etc/init.d/iptables stop

# chkconfig-- level 345 iptables off

(2) close seliux

# vim / etc/sysconfig/selinux

SELINUX=disabled

(3) ip corresponds to hostname

# vim / etc/hosts

172.16.101.54 sht-sgmhadoopcm-01

172.16.101.55 sht-sgmhadoopnn-01

172.16.101.56 sht-sgmhadoopnn-02

(4) install JDK

# tar xf / usr/java/jdk-8u111-linux-x64.tar.gz

# chown-R root:root jdk1.8.0_111

# vim / etc/profile

JAVA_HOME=/usr/java/jdk1.8.0_111/

PATH=$JAVA_HOME/bin:$PATH

CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

Export JAVA_HOME

Export CLASSPATH

Export PATH

# source / etc/profile

(5) check the python version

# python-V

Python 2.7.5

(6) NTP synchronization

If your environment is your own virtual machine, you need to configure ntp services to ensure clock synchronization on each node.

If it is a cloud environment, no configuration is required.

(7) close large pages

Echo never > / sys/kernel/mm/redhat_transparent_hugepage/defrag

Echo never > / sys/kernel/mm/redhat_transparent_hugepage/enabled

# cat / etc/rc.local

Echo never > / sys/kernel/mm/redhat_transparent_hugepage/defrag

Echo never > / sys/kernel/mm/redhat_transparent_hugepage/enabled

(8) set swap

# vim / etc/sysctl.conf

Vm.swappiness = 0

# sysctl-p

4.CM installation configuration (configured only on cm management nodes)

(1) configure http service

Yum install-y httpd

/ etc/init.d/httpd start

Browser access: http://172.16.101.54

(2) download and upload the parcel file of CDH5.10.0: http://archive.cloudera.com/cdh6/parcels/5.10.0/

# Note: CDH-5.10.0-1.cdh6.10.0.p0.41-el6.parcel.sha1 needs to be renamed to CDH-5.10.0-1.cdh6.10.0.p0.41-el6.parcel.sha

# ll / var/www/html/cdh_parcel

-rwxr-xr-x 1 root root 1501694035 Feb 20 15:17 CDH-5.10.0-1.cdh6.10.0.p0.41-el6.parcel

-rwxr-xr-x 1 root root 41 Feb 20 15:06 CDH-5.10.0-1.cdh6.10.0.p0.41-el6.parcel.sha

-rwxr-xr-x 1 root root 64807 Jan 26 2017 manifest.json

(3) download and upload the RPM package of CDH5.10.0: cm5.10.0-centos6.tar.gz

Http://archive.cloudera.com/cm5/repo-as-tarball/5.10.0/

# mkdir-p / var/www/html/cm5/redhat/6/x86_64/

# tar-zxvf cm5.10.0-centos6.tar.gz-C / var/www/html/cm5/redhat/6/x86_64/

# chmod-R 755 / var/www/html/*

Browser access: http://172.16.101.54/cdh_parcel/

Browser access: http://172.16.101.54/cm5/redhat/6/x86_64/cm/5.10.0/RPMS/x86_64/

5. Configure the yum source (on each node)

# vim / etc/yum.repos.d/cloudera-manager.repo

[cloudera-manager]

Name = Cloudera Manager, Version 5.10.0

Baseurl = http://172.16.101.54/cm5/redhat/6/x86_64/cm/5/

Gpgcheck = 0

6. Install the RPM package and install the configuration mysql database (operate on cm nodes only)

(1) install CM

# yum clean all

# cd / var/www/html/cm5/redhat/6/x86_64/cm/5.10.0/RPMS/x86_64/

# yum install-y cloudera-manager-daemons-5.10.0-1.cm5100.p0.85.el6.x86_64.rpm

# yum install-y cloudera-manager-server-5.10.0-1.cm5100.p0.85.el6.x86_64.rpm

(2) configuration database

# vim / etc/cloudera-scm-server/db.properties

Com.cloudera.cmf.db.type=mysql

Com.cloudera.cmf.db.host=172.16.101.54:3306

Com.cloudera.cmf.db.name=cmf

Com.cloudera.cmf.db.user=cmf

Com.cloudera.cmf.db.password=cmf_password

Com.cloudera.cmf.db.setupType=EXTERNAL

(3) download the mysql jdbc driver package

# cd / usr/share/java

# wget http://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.37.zip

# unzip mysql-connector-java-5.1.37.zip

# cd mysql-connector-java-5.1.37

# cp mysql-connector-java-5.1.37-bin.jar.. / mysql-connector-java.jar

(4) omit the process of installing mysql

# create libraries and users

Create database cmf DEFAULT CHARACTER SET utf8

Grant all on cmf.* TO 'cmf'@'%' IDENTIFIED BY' cmf_password'

Create database amon DEFAULT CHARACTER SET utf8

Grant all on amon.* TO 'amon'@'%' IDENTIFIED BY' amon_password'

Flush privileges

(5) start the cm service, which takes some time, about 3minutes, to create a series of tables in mysql

If startup fails, you can check the log of / var/log/cloudera-scm-server/

# / etc/init.d/cloudera-scm-server start

7. Graphical wizard installation

Browser access: http://172.16.101.54:7180/

Both user name and password are admin

(1) choose the version to be installed: Cloudera Express

(2) data IP address search: 172.16.101. [54-56]

(3) choose the installation method:

a. Using the parcel package, more:

Parcel directory: / opt/cloudera/parcels

(that is, the installation directory of CDH software, each node will be created automatically, and after installation, about 3.5G)

Local parcel repository directory: / opt/cloudera/parcel-repo

Remote Parcel repository URL: http://172.16.101.54/cdh_parcel/

(that is, the http directory where the parcel package is stored)

b. Select the specific distribution of Cloudera Manager Agent that you want to install on the host: (note that the http here is not written as https, otherwise an error will be reported later)

Custom repository: http://172.16.101.54/cm5/redhat/6/x86_64/cm/5/

(4) JDK installation option-do not check

(5) enable when user mode-do not check

(6) users who use root users or can sudo to root without a password

Users who use LDAP here need to configure it on each node in advance:

# visudo

Xxxuser ALL= (ALL) NOPASSWD:ALL

(7) Cluster installation

Probably need 20minutes

This step is mainly to install cloudera-scm-agent,cloudera-manager-daemons on each node and start agent.

(8) Cluster settings

Select the service portfolio to install: custom service HDFS,YARN,Zookeeper

(9) Custom role assignment, and "2. Role Planning" same

(10) Database settings

Database hostname: 172.16.101.54purl 3306

Database type: MySQL

Library: amon

User: amon

Password: amon_password

Test connection succeeded, next

(11) Review changes: (no modification is required, all can be done by default)

HDFS block size 128

Accepted DataNode failed volume: 0

DataNode data directory / dfs/dn

NameNode data directory / dfs/nn

HDFS checkpoint directory / dfs/snn

Host Monitor storage directory / var/lib/cloudera-host-monitor

Service Monitor storage directory / var/lib/cloudera-service-monitor

NodeManager local directory / yarn/nm

Data directory / var/lib/zookeeper

Transaction log directory / var/lib/zookeeper

(12) installation completed

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.