Shulou(Shulou.com)06/03 Report--

Installation and configuration of CDH5.13.3 under centos 6.8 (detailed deployment)

First, prepare before installation

1.1. Download address

Wget http://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.13.3_x86_64.tar.gz

Http://archive.cloudera.com/cdh6/parcels/5.13.3/

Wget http://archive.cloudera.com/cdh6/parcels/5.13.3/manifest.json

Wget http://archive.cloudera.com/cdh6/parcels/5.13.3/CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel

Wget http://archive.cloudera.com/cdh6/parcels/5.13.3/CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1

1.2. Firewall is off

/ etc/init.d/iptables stop

Chkconfig iptables off

1.3.Setting selinux

Vi / etc/selinux/config

1.4.Setting time synchronization ntp (per machine)

Yum-y install ntpdate

Crontab-e

* / 5 * ntpdate 0.rhel.pool.ntp.org

Save exit

Vi / etc/hosts (increased hadoop01)

Note: 127.0.0.1 hadoop01 is required for Cloudera-Manager.

127.0.0.1 hadoop01

192.168.1.201 hadoop01

192.168.1.202 hadoop02

192.168.1.203 hadoop03

Vi / etc/hosts (other two hadoo02, hadoop03)

192.168.1.201 hadoop01

192.168.1.202 hadoop02

192.168.1.203 hadoop03

Dependent package installation (per machine)

Yum-y install chkconfig python bind-utils psmisc libxslt zlib sqlite fuse fuse-libs redhat-lsb cyrus-sasl-plain cyrus-sasl-gssapi openssh-clients

3. Jdk installation (per machine)

Tar zxvf jdk-8u111-linux-x64.gz

Mv jdk1.8.0_111 / usr/local/jdk

Configure environment variables

Vi / etc/profile

# # jdk

Export JAVA_HOME=/usr/local/jdk

Export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

Export PATH=$JAVA_HOME/bin:$PATH

Let environmental variables produce

Source / etc/profile

IV. Ssh password-free login configuration

Ssh-keygen-t rsa

Ssh-copy-id 192.168.1.201

Ssh-copy-id 192.168.1.202

Ssh-copy-id 192.168.1.202

Install mysql (install on haoop01)

5.1. Install the dependency package

Yum-y install libaio*

Yum-y install libnuma*

5.2.Create mysql users

Useradd mysql

5.3. Create a directory

Mkdir-p / data/app

Mkdir / data/app/mysql3306

Mkdir / data/app/undolog

5.4. Decompress the file

Tar zxvf mysql-5.6.41-linux-glibc2.12-x86_64.tar.gz

Mv mysql-5.6.41-linux-glibc2.12-x86_64 / data/app/mysql5.6

Mkdir / data/app/mysql5.6/logs

5.5.Configuring my.cnf

Vi / data/app/mysql5.6/my.cnf

[client]

Socket=/data/app/mysql5.6/mysqld.sock

Port=3306

[mysql]

Prompt=\\ u@\\ d\ r:\\ m:\\ s >

No-auto-rehash

Default-character-set=utf8

[mysqld_safe]

Log-error=/data/app/mysql3306/mysqld.error

[mysqld]

Socket=/data/app/mysql5.6/mysqld.sock

Pid-file=/data/app/mysql5.6/mysqld.pid

Basedir=/data/app/mysql5.6

Datadir=/data/app/mysql3306

Port=3306

Server-id=1201

Character-set-server=utf8

Skip-external-locking

# skip-name-resolve

Max_connections=1000

Max_connect_errors=1000

Max_allowed_packet=128M

Table_definition_cache=500

Table_open_cache=500

Sort_buffer_size = 1m

Tmp_table_size = 64m

Read_buffer_size = 1m

Read_rnd_buffer_size = 1m

Myisam_sort_buffer_size = 64m

Thread_cache_size = 8

Query_cache_type=0

Query_cache_size=0

Lower_case_table_names = 1

Log_bin_trust_function_creators = 1

# slow log#

Slow-query_log=1

Slow-query_log_file=/data/app/mysql5.6/logs/mysql.slow

Long_query_time=2

# binlog##

Log-bin=mysql-bin

Binlog-format=ROW

Expire_logs_days=10

Log-slave-updates=1

Sync_binlog=0

Auto_increment_increment=2

Auto_increment_offset=1

# # #

# # #

# # #

Innodb_undo_directory=/data/app/undolog

Innodb_undo_logs = 128,

Innodb_undo_tablespaces = 3

# INNODB#

Transaction-isolation=READ-COMMITTED

Innodb_buffer_pool_size=2G

Innodb_flush_log_at_trx_commit=2

Innodb_strict_mode=1

Innodb_flush_method=O_DIRECT

Innodb_file_format=Barracuda

Innodb_log_files_in_group=3

Innodb_file_per_table=1

Innodb_io_capacity=500

Innodb_support_xa=1

[mysqldump]

Quick

Max_allowed_packet=128M

[mysqlhotcopy]

Interactive_timeout = 28400

5.6. Increase permissions

Chown-R mysql:mysql / data/app

5.7. Configure environment variables

Vi / etc/profile

# # mysql

Export MYSQL_HOME=/data/app/mysql5.6

Export MY_BASEDIR_VERSION=/data/app/mysql5.6

Export PATH=/data/app/mysql5.6/bin:$PATH

Export LD_LIBRARY_PATH=/data/app/mysql5.6/lib

Environmental entry into force

Source / etc/profile

5.8. Initialize the database

/ data/app/mysql5.6/scripts/mysql_install_db-user=mysql-basedir=/data/app/mysql5.6-datadir=/data/app/mysql3306

5.9. Configure the startup file

Cp / data/app/mysql5.6/support-files/mysql.server / etc/init.d/mysqld

Vi / etc/init.d/mysqld

Basedir=/data/app/mysql5.6

Datadir=/data/app/mysql3306

Save exit

Chmod + x / etc/init.d/mysqld

Chkconfig mysqld on

Start

/ etc/init.d/mysqld start

5.10. Initialize password

/ data/app/mysql5.6/bin/mysqladmin-u root password 'yb2018#08'

5.11. Set remote permissions

# mysql-uroot-pyb2018#08

Warning: Using a password on the command line interface can be insecure.

Welcome to the MySQL monitor. Commands end with; or\ g.

Your MySQL connection id is 2

Server version: 5.6.41-log MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

Affiliates. Other names may be trademarks of their respective

Owners.

Type 'help;' or'\ h' for help. Type'\ c'to clear the current input statement.

Root@ (none) 10:28:44 > grant all on *. * to 'scm'@'localhost' identified by' temp' with grant option

Query OK, 0 rows affected (0.00 sec)

Root@ (none) 10:28:58 > grant all on *. * to 'cdh'@'%' identified by' cdh#2018' with grant option

Query OK, 0 rows affected (0.00 sec)

Root@ (none) 10:29:08 > create database hive

Query OK, 1 row affected (0.00 sec)

Root@ (none) 10:29:16 > create database ooz

Query OK, 1 row affected (0.00 sec)

Root@ (none) 10:29:27 > create database hue

Query OK, 1 row affected (0.00 sec)

Root@ (none) 10:29:36 > flush privileges

Query OK, 0 rows affected (0.00 sec)

6. Install Cloudera-Manager

6.1. Extract the package to the specified directory, and all servers need to

Mkdir-p / data/cloudera-manager

Tar zxvf cloudera-manager-el6-cm5.13.3_x86_64.tar.gz-C / data/cloudera-manager

6.2 create cloudera-scm users (all nodes)

Useradd-r-d / data/cloudera-manager/cm-5.13.3/run/cloudera-scm-server-M-c "Cloudera SCM User" cloudera-scm

Id cloudera-scm

Configure the slave node cloudera-manger-agent to point to the master node server (hadoop02,hadoop03 configuration)

Vi / data/cloudera-manager/cm-5.13.3/etc/cloudera-scm-agent/config.ini

Change server_host to the hostname where CMS is located, namely hadoop01

Create a parcel-repo repository directory in the primary node (this is only required on the primary node hadoop01)

Mkdir-p / data/cloudera/parcel-repo

Chown cloudera-scm:cloudera-scm / data/cloudera/parcel-repo

Cp CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1 manifest.json / data/cloudera/parcel-repo/

Cd / data/cloudera/parcel-repo/

Mv CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1 CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha

Note: 1 should be removed from the CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1 suffix.

6.5.All nodes create parcels directories (all nodes)

Mkdir-p / data/cloudera/parcels

Chown cloudera-scm:cloudera-scm / data/cloudera/parcels

Explanation: Clouder-Manager extracts CDHs from the / data/cloudera/parcel-repo directory of the master node, distributes and decompresses it and activates it into the / data/cloudera/parcels directory of each node

6.6.The initial script configuration database scm_prepare_database.sh (on the primary node)

/ data/cloudera-manager/cm-5.13.3/share/cmf/schema/scm_prepare_database.sh mysql-uscm-ptemp scm scm temp

If an error is reported, the operation

Mkdir / usr/share/java

Cp mysql-connector-java-5.1.46-bin.jar / usr/share/java/mysql-connector-java.jar

The above picture shows the success.

Successfully comment out # 127.0.0.1 localhost hadoop01, or there will be a problem later.

Vi / etc/hosts

# 127.0.0.1 hadoop01

192.168.1.201 hadoop01

192.168.1.202 hadoop02

192.168.1.203 hadoop03

6.7. start the primary node cloudera-scm-server (hadoop01 operation)

Cp / data/cloudera-manager/cm-5.13.3/etc/init.d/cloudera-scm-server / etc/init.d/cloudera-scm-server

Chmod + x / etc/init.d/cloudera-scm-server

Start

/ etc/init.d/cloudera-scm-server start

If an error is reported at File not found: / usr/sbin/cmf-server

Vi / etc/init.d/cloudera-scm-server

Find CMF_DEFAULTS=$ {CMF_DEFAULTS:-/etc/default}

Modify to CMF_DEFAULTS=/data/cloudera-manager/cm-5.13.3/etc/default

Start

/ etc/init.d/cloudera-scm-server start

6.8. Start all cloudera-scm-agent nodes (hadoop02,hadoop03 operation)

Mkdir / data/cloudera-manager/cm-5.13.3/run/cloudera-scm-agent

Cp / data/cloudera-manager/cm-5.13.3/etc/init.d/cloudera-scm-agent / etc/init.d/cloudera-scm-agent

Chmod + x / etc/init.d/cloudera-scm-agent

/ etc/init.d/cloudera-scm-agent start

Vi / etc/init.d/cloudera-scm-agent

Find CMF_DEFAULTS=$ {CMF_DEFAULTS:-/etc/default}

Modify to CMF_DEFAULTS=/data/cloudera-manager/cm-5.13.3/etc/default

Start

/ etc/init.d/cloudera-scm-agent start

6.9. Login user's admin password admin

Http://192.168.1.201:7180

Go ahead

Choose to be free and continue

Go ahead

Select the current management host, select hadoop02,hadoop03, and continue

Click more options to configure

After the changes are made, save the changes

Come out after restart as shown in the following figure

/ etc/init.d/cloudera-scm-server restart

Restart before continuing as shown above, because the default / opt installation

Go ahead

Each one performs the following

Echo 10 > / proc/sys/vm/swappiness

Echo never > / sys/kernel/mm/transparent_hugepage/defrag

Echo never > / sys/kernel/mm/transparent_hugepage/enabled

Continue to choose different points according to your own needs.

Modify two of the above Datanode

Enter the connection host name, database name, user name and password. Click to test the connection after Succesful is successful, and click continue.

Click to continue according to your own modification

Go ahead

Installation completed

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.