Shulou(Shulou.com)06/03 Report--

1. Software environment and IP planning

RHEL6 role

Jdk-8u45apache-maven-3.3.9

Hive-1.1.0-cdh6.7.1-src.tar.gz

Hadoop-2.8.1.tar.gz

Mysql-connector-java-6.0.6.tar.gz

Apache-maven-3.3.9

Cloudera-manager-el6-cm5.9.3_x86_64.tar

Mysql-5.7

CDH-5.9.3-1.cdh6.9.3.p0.4-el6

172.16.18.133 NN & & SN & & Jobtrack hadoop01

172.16.18.134 DN & & tasktrack hadoop02

172.16.18.136 DN & & tasktrack hadoop03

172.16.18.143 DN & & tasktrack hadoop04

172.16.18.145 DN & & tasktrack hadoop05

NN = namenode SN=secondarynamenode DN=datanode

Cluster introduction:

There are mainly three free Hadoop versions (all foreign manufacturers), namely: Apache (the most original version, all distributions are improved based on this version), Cloudera version (Cloudera's Distribution Including Apache Hadoop, referred to as CDH), Hortonworks version (Hortonworks Data Platform, referred to as "HDP"), for the domestic, the vast majority choose the CDH version.

CDH (Cloudera's Distribution, including Apache Hadoop), one of the many branches of Hadoop, is maintained by Cloudera, built on a stable version of Apache Hadoop, and integrates many patches that can be used directly in production environments.

Cloudera Manager is to facilitate the installation, monitoring and management of related services such as Hadoop in the cluster, which greatly simplifies the installation and configuration management of hosts, Hadoop, Hive, Spark and other services in the cluster.

For cluster installation, this article chooses to install CDH offline.

Https://www.cloudera.com/downloads/cdh/5-9-0.html

Description of CDH on the official website, CHD support list for system JDK database and other versions

The official website supports jdk1.8, but some 1.8 versions will report errors, so we choose jdk1.7.

2. Software package equipment

Cloudera Manager software package

Http://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.9.3_x86_64.tar.gz

CDH package (download the corresponding Linux version package)

Http://archive.cloudera.com/cdh6/parcels/5.9.3/CDH-5.9.3-1.cdh6.9.3.p0.4-el6.parcel

Http://archive.cloudera.com/cdh6/parcels/5.9.3/CDH-5.9.3-1.cdh6.9.3.p0.4-el6.parcel.sha1

The version of the mysql jdbc driver is:

Http://download.softagency.net/MySQL/Downloads/Connector-J/mysql-connector-java-6.0.6.tar.gz

3. System-related configuration

All hosts have the same installation JDK shutdown selinux iptables configuration / etc/hosts configuration yum

[root@hadoop01 ~] # vim / etc/profile

Export JAVA_HOME=/usr/java/jdk1.7.0_79

Export PATH=$JAVA_HOME/bin:$ORACLE_HOME/bin:$R_HOME/bin:$PATH

[root@hadoop01 ~] # getenforce

Disabled

[root@hadoop01] # iptables-L

Chain INPUT (policy ACCEPT)

Target prot opt source destination

Chain FORWARD (policy ACCEPT)

Target prot opt source destination

Chain OUTPUT (policy ACCEPT)

Target prot opt source destination

[root@hadoop01 ~] # cat / etc/hosts

127.0.0.1 localhost

172.16.18.133 hadoop01

172.16.18.134 hadoop02

172.16.18.136 hadoop03

172.16.18.143 hadoop04

172.16.18.145 hadoop05

[root@hadoop01 ~] # hostname

Hadoop01

4. Configure ssh automatic login mutual trust

Refer to pseudo-distributed ssh mutual trust configuration

Each node verification does not require interactive input of yes

Useradd hadoop-set up users

Ssh hadoop01 date

Ssh hadoop02 date

Ssh hadoop03 date

Ssh hadoop04 date

Ssh hadoop05 date

5. Modify the swappiness=0 of swap space

Cat / proc/sys/vm/swappiness

Sysctl vm.swappiness=0

Echo 0 > / proc/sys/vm/swappiness

Turn off alarm: echo never > / sys/kernel/mm/transparent_hugepage/defrag

6. Configure the NTP server

Select the master server first, and all other servers synchronize the time of this master server.

# hwclock-w

Configure boot boot

[root@hadoop01 ~] # chkconfig ntpd on

[root@hadoop01] # chkconfig-- list ntpd

[root@hadoop01 ~] # vi / etc/ntp.conf

(find this line, release the comment on restrict, and change the ip address)

# Hosts on local network are less restricted.

Restrict 192.168.128.1 mask 255.255.255.0 nomodify notrap

(find this line and comment on the server below)

# Please consider joining the pool (http://www.pool.ntp.org/join.html).

# server 0.rhel.pool.ntp.org iburst

# server 1.rhel.pool.ntp.org iburst

# server 2.rhel.pool.ntp.org iburst

# server 3.rhel.pool.ntp.org iburst

Add the following two lines

Server 127.0.1.0 # local clock

Fudge 127.0.1.0 stratum 10

Configure other servers

Vi / etc/ntp.conf

# Hosts on local network are less restricted.

Restrict 192.168.128.1 nomodify notrap noquery

(note below server)

# Please consider joining the pool (http://www.pool.ntp.org/join.html).

# server 0.rhel.pool.ntp.org iburst

# server 1.rhel.pool.ntp.org iburst

# server 2.rhel.pool.ntp.org iburst

# server 3.rhel.pool.ntp.org iburst

Specified time service

Server 192.168.128.51

All restart ntp services

[root@hadoop01 ~] # service ntpd restart

[root@hadoop01 ~] # ntpstat

Synchronised to NTP server (172.16.18.33) at stratum 12

Time correct to within 18 ms

Polling server every 64 s

[root@hadoop01 ~] # date

7. Disable ipv6 and "transparent large pages"

[root@hadoop01 ~] # echo "alias ipv6 off" > > / etc/modprobe.d/dist.conf

[root@hadoop01 ~] # echo "alias net-pf-10 off" > > / etc/modprobe.d/dist.conf

[root@hadoop01 ~] # echo never > / sys/kernel/mm/transparent_hugepage/defrag

[root@hadoop01 ~] # echo 'echo never > / sys/kernel/mm/transparent_hugepage/defrag' > > / etc/rc.local

[root@hadoop01 ~] #

8. Prepare the mysql database

Modify mysql permissions:

GRANT ALL PRIVILEGES ON *. * TO 'root'@'%' IDENTIFIED BY' 123' WITH GRANT OPTION

Flush privileges

Delete from user where host! ='%'

[root@hadoop01 software] # mysql-h 172.16.18.133-uroot-p

# # preparation # #

Hadoop01 Server | | Agent

Hadoop02 Agent

Hadoop03 Agent

CDH uses 3 servers, and the remaining 2 are used as clusters to add nodes.

#

10.CM installation

Install cloudera Manager Server, Agent

All cdh cluster nodes need to install software to prepare for account establishment.

Primary node:

[root@hadoop01 software] # ls cloudera-manager-el6-cm5.9.3_x86_64.tar.gz

Cloudera-manager-el6-cm5.9.3_x86_64.tar.gz

[root@hadoop01 software] # pwd

/ opt/software

[root@hadoop01 software] # ls cloudera-manager-el6-cm5.9.3_x86_64.tar.gz

Cloudera-manager-el6-cm5.9.3_x86_64.tar.gz

[root@hadoop01 software] # mkdir / opt/cloudera-manager

[root@hadoop01 software] # tar zxvf cloudera-manager-el6-cm5.9.3_x86_64.tar.gz-C / opt/cloudera-manager/

Client configuration

/ opt/cloudera-manager/cm-5.9.3/etc/cloudera-scm-agent/config.ini

Server_host=hadoop01-in cm server hostname

[root@hadoop01 software] # useradd-system-no-create-home-shell=/bin/false-comment "Cloudera SCM User" cloudera-scm

[root@hadoop01 software] # id cloudera-scm

Uid=495 (cloudera-scm) gid=492 (cloudera-scm) group = 492 (cloudera-scm)

Haoop02 hadoop03 all slave nodes

Useradd-system-no-create-home-shell=/bin/false-comment "Cloudera SCM User" cloudera-scm

Mkdir / opt/cloudera-manager

[root@hadoop01 opt] # scp-r / opt/cloudera-manager/cm-5.9.3 hadoop02:/opt/cloudera-manager/

[root@hadoop01 opt] # scp-r / opt/cloudera-manager/cm-5.9.3 hadoop03:/opt/cloudera-manager/

11. Configure CM Server database

We began to prepare for the establishment of the mysql database

[root@hadoop01] # mysql-h272.16.18.133-uroot-p

Mysql >

GRANT ALL PRIVILEGES ON *. * TO 'root'@'%' IDENTIFIED BY' 123' WITH GRANT OPTION

Flush privileges

Mysql > flush privileges

[root@hadoop01 schema] # pwd

/ opt/cloudera-manager/cm-5.9.3/share/cmf/schema

[root@hadoop01 schema] #. / scm_prepare_database.sh mysql-hhadoop01-uroot-p123-- scm-host hadoop01 cmdb root 123

JAVA_HOME=/usr/java/jdk1.7.0_79

Verifying that we can write to / opt/cloudera-manager/cm-5.9.3/etc/cloudera-scm-server

Sat Apr 28 14:20:38 CST 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45mm, 5.6.26 + and 5.7.6 + requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.

Creating SCM configuration file in / opt/cloudera-manager/cm-5.9.3/etc/cloudera-scm-server

Executing: / usr/java/jdk1.7.0_79/bin/java-cp / usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera-manager/cm-5.9.3/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor / opt/cloudera-manager/cm-5.9.3/etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.

Sat Apr 28 14:20:39 CST 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45mm, 5.6.26 + and 5.7.6 + requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.

[main] DbCommandExecutor INFO Successfully connected to database.

All done, your SCM database is configured correctly!

Description: this script is used to create and configure the database required by CMS. The parameters refer to:

Mysql: the database uses mysql, so if you use oracle during installation, this parameter should be changed to oracle.

-hadoop01: the database is built on the hadoop01 host, that is, the primary node.

-run mysql as uroot:root. -the root password for 123:mysql is

-- scm-host hadoop01: the host of CMS, usually on the same host as the host installed by mysql

The last three parameters are: database name, database user name, database password.

12. Create the local source of CDH

Server node

Mkdir-p / opt/cloudera/parcel-repo

Chown cloudera-scm:cloudera-scm / opt/cloudera/parcel-repo

Agent node

Mkdir-p / opt/cloudera/parcels

Chown cloudera-scm:cloudera-scm / opt/cloudera/parcels

Upload to CDH-5.9.3-1.cdh6.9.3.p0.4-el6.parcel manifest.json master node / opt/cloudera/parcel-repo/ path

[root@hadoop01 CDH] # cd / opt/cloudera/parcel-repo/

[root@hadoop01 parcel-repo] # ls

CDH-5.9.3-1.cdh6.9.3.p0.4-el6.parcel manifest.json

[root@hadoop01 parcel-repo] # ls

CDH-5.9.3-1.cdh6.9.3.p0.4-el6.parcel manifest.json

[root@hadoop01 parcel-repo] # mv manifest.json CDH-5.9.3-1.cdh6.9.3.p0.4-el6.parcel.sha

Manifest.json rename the file name is the same as your parel package name, with the .sha suffix

13. Start

Ensure that mysql starts first.

Server:hadoop01

[root@hadoop01 init.d] # pwd

/ opt/cloudera-manager/cm-5.9.3/etc/init.d

[root@hadoop01 init.d] #. / cloudera-scm-server start

Starting cloudera-scm-server:

Agent:hadoop01 hadoop02 hadoop02

/ opt/cloudera-manager/cm-5.9.3/etc/init.d

. / cloudera-scm-agent start

Starting cloudera-scm-agent: [OK]

2018-04-28 14 43 37022 INFO WebServerImpl:org.mortbay.log: jetty-6.1.26.cloudera.4

2018-04-28 14 43 37 024 INFO WebServerImpl:org.mortbay.log: Started SelectChannelConnector@0.0.0.0:7180

2018-04-28 14 43 37 024 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

The following indicates that the startup is successful

14. Graphics access

A complete list of errors:

Problem 1:JDBC driver driver

[root@hadoop01 schema] #. / scm_prepare_database.sh mysql cmdb-h hadoop01-uroot-p123456-- scm-host hadoop01 scm scm scm

JAVA_HOME=/usr/java/jdk1.7.0_79

Verifying that we can write to / opt/cloudera-manager/cm-5.9.3/etc/cloudera-scm-server

[main] DbProvisioner ERROR Unable to find the MySQL JDBC driver. Please make sure that you have installed it as per instruction in the installation guide.

[main] DbProvisioner ERROR Stack Trace:

Java.lang.ClassNotFoundException: com.mysql.jdbc.Driver

At java.net.URLClassLoader$1.run (URLClassLoader.java:366) [: 1.7.0,79]

At java.net.URLClassLoader$1.run (URLClassLoader.java:355) [: 1.7.0,79]

At java.security.AccessController.doPrivileged (Native Method) [: 1.7.0,79]

At java.net.URLClassLoader.findClass (URLClassLoader.java:354) [: 1.7.0,79]

Solution method

[root@hadoop01 software] # ls mysql-connector-java-5.1.46.zip

Mysql-connector-java-5.1.46.zip

[root@hadoop01 software] # unzip mysql-connector-java-5.1.46.zip ^ C

[root@hadoop01 software] # cp mysql-connector-java-5.1.46/

Build.xml mysql-connector-java-5.1.46-bin.jar README.txt

CHANGES mysql-connector-java-5.1.46.jar src/

COPYING README

[root@hadoop01 software] # cp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46.jar / usr/share/java/

[root@hadoop01 software] # mv / usr/share/java/mysql-connector-java-5.1.46.jar / usr/share/java/mysql-connector-java.jar

Question 2:

Dbc url 'jdbc:mysql://hadoop01/?useUnicode=true&characterEncoding=UTF-8'

Java.sql.SQLException: Access denied for user 'root'@'hadoop01' (using password: YES)

At com.mysql.jdbc.SQLError.createSQLException (SQLError.java:965) [mysql-connector-java.jar:5.1.46]

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.