Shulou(Shulou.com)06/03 Report--

Detailed installation steps for CDH6.0.1

System environment

Operating system: 3 CentOS 7.3x64

Cloudera Manager:6.0.1

CDH: 6.0.1

VM NameVM IP AdressCentos_HostNameRoleCentos_wjg1192.168.1.101hadoop01Cloudera Manager Server + Cloudera Manager AgentCentos_wjg2192.168.1.102hadoop02Cloudera Manager AgentCentos_wjg6192.168.1.106hadoop06Cloudera Manager Agent

I. configure the network environment

Edit the / etc/sysconfig/network-scripts/ifcfg-eth0 configuration file to modify the items indicated in the following red colors:

Vi / etc/sysconfig/network-scripts/ifcfg-eth0

TYPE=Ethernet

BOOTPROTO=static

DEFROUTE=yes

PEERDNS=yes

PEERROUTES=yes

IPV4_FAILURE_FATAL=no

IPV6INIT=yes

IPV6_AUTOCONF=yes

IPV6_DEFROUTE=yes

IPV6_PEERDNS=yes

IPV6_PEERROUTES=yes

IPV6_FAILURE_FATAL=no

NAME=eth0

UUID=3719cf03-5d05-454b-903f-e1e26ad34fd6

DEVICE=eth0

ONBOOT=yes

IPADDR=192.168.1.101

NETWORK=192.168.1.1

NETSTAT=255.255.255.0

GATEWAY=192.168.1.1

DNS1=222.186.160.70

DNS2=114.114.114.114

two。 Edit the / etc/default/grub file, in the "GRUB_CMD_LINE_LINUX=" item, insert "net.ifnames=0 biosdevname=0" (separated by a space from the original item), and save the exit.

3. Call the command grub2-mkconfig-o / boot/grub2/grub.cfg to update the configuration, and then restart directly with reboot.

After that, when you check it with the ifconfig-a command, there will be an eth0 network card. Change hostname

Hostnamectl set-hostname hadoop01 modifies the hosts file

Vi / etc/hosts

192.168.1.101 hadoop01

192.168.1.102 hadoop02

192.168.1.106 hadoop06

Turn off the firewall

Sed-I's etc/selinux/config SelinuxplenforcingAccording to SELINUXAfter

Setenforce 0

Systemctl stop firewalld.service

Systemctl disable firewalld.service

Configure secret-free login:

Generate rsa public and private keys on each host

Ssh-keygen-t rsa

Cd .ssh

Cat id_rsa.pub > > authorized_keys

Log in to other hosts and copy the contents of the public key files of other hosts to the authorized_keys file on the hadoop01 host. The command is as follows:

[root@hadoop06 .ssh] # ssh-copy-id-I hadoop01

/ usr/bin/ssh-copy-id: INFO: attempting to log in with the new key (s), to filter out any that are already installed

/ usr/bin/ssh-copy-id: INFO: 1 key (s) remain to be installed-- if you are prompted now it is to install the new keys

Wjg@hadoop01's password:

Number of key (s) added: 1

Now try logging into the machine, with: "ssh 'hadoop01'"

And check to make sure that only the key (s) you wanted were added.

Repeat the above steps on each host so that any two hosts can connect to each other.

At this point, password-free login has been set up.

Change the system time:

Go to the home directory and modify .bash _ profile

# setup time region

TZ='Asia/Shanghai'

Export TZ

Or: tzselect

Write the current time and date to BIOS to avoid invalidation after restart

Hwclock-w

Configure the time for NTP to synchronize each server

9.1 Primary Node configuration

Before configuration, use ntpdate to manually synchronize the time, lest the time gap between the local machine and the timing center is too big, so that the ntpd can not be synchronized normally. Here we choose 65.55.56.206 as the timing center, ntpdate-u ntp1.aliyun.com

Vi / etc/ntp.conf

# Hosts on local network are less restricted.

Uncomment the following line and change the IP and mask to real-world IP and mask, which is configured to allow ntp client connections

Restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

Locate server 0.centos.pool.ntp.org iburst and annotate all server configurations

Add the following two lines

Server 127.127.1.0 # local clock

Fudge 127.127.1.0 stratum 10

Start the NTP service

After setting up, restart the ntpd service and use ntpstat to check the effect [you can see the effect in a few seconds]

# systemctl restart ntpd

# ntpstat

Vi / etc/sysconfig/ntpdate

# Set to 'yes' to sync hw clock after successful ntpdate

SYNC_HWCLOCK=yes

9.2 Agent node configuration

Modify / etc/ntp.conf

Make the following changes to the file:

1. Annotate all restrict and server configurations

2. To add the following note, you need to modify the following IP to be the IP of the NTP server (in this case, the IP of the server host)

Server 192.168.1.100

Set the ntp service of all hosts to boot and start automatically

Chkconfig ntpd on

Systemctl enable ntpd.service

2. Configure 163yum source

1) download repo file

Cd / etc/yum.repos.d/

Wget http://mirrors.163.com/.help/CentOS7-Base-163.repo

2) back up and replace the repo files of the system

Mv CentOS-Base.repo CentOS-Base.repo.bak

Cp CentOS7-Base-163.repo CentOS-Base.repo

3) execute yum source update command

Yum clean all

Yum makecache

Yum update

Yum install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb-y

Yum install httpd mod_ssl openssl-devel python-psycopg2-y

Yum install MySQL-python

Yum install perl

Yum install autoconf

3. Install MySQL:

View and uninstall MySQL and Mariadb that come with CentOS system

Rpm-qa | grep MySQL

Rpm-qa | grep mariadb

Rpm-e-- nodeps MySQL-python-1.2.5-1.el7.x86_64

Rpm-e-- nodeps mariadb-libs-5.5.60-1.el7_5.x86_64

Download and upload the following rpm package to the master node

MySQL-client-5.6.42-1.el7.x86_64.rpm

MySQL-devel-5.6.42-1.el7.x86_64.rpm

MySQL-server-5.6.42-1.el7.x86_64.rpm

On the primary node, execute the following command to install MySQL

Rpm-ivh MySQL*

Start MySQL after the installation is complete:

Service mysql start

Systemctl start mysql.service

Modify the configuration file of MySQL

Vi / etc/my.cnf

[mysqld]

Datadir=/var/lib/mysql

Socket=/var/lib/mysql/mysql.sock

Transaction-isolation = READ-COMMITTED

# Disabling symbolic-links is recommended to prevent assorted security risks

# to do so, uncomment this line:

Symbolic-links = 0

Key_buffer_size = 32m

Max_allowed_packet = 32m

Thread_stack = 256K

Thread_cache_size = 64

Query_cache_limit = 8m

Query_cache_size = 64m

Query_cache_type = 1

Max_connections = 5505

# expire_logs_days = 10

# max_binlog_size = 100m

# log_bin should be on a disk with enough free space.

# Replace'/ var/lib/mysql/mysql_binary_log' with an appropriate path for your

# system and chown the specified folder to the mysql user.

Log_bin=/var/lib/mysql/mysql_binary_log

# In later versions of MySQL, if you enable the binary log and do not set

# a server_id, MySQL will not start. The server_id must be unique within

# the replicating group.

Server_id=1

Binlog_format = mixed

Read_buffer_size = 2m

Read_rnd_buffer_size = 16m

Sort_buffer_size = 8m

Join_buffer_size = 8m

# InnoDB settings

Innodb_file_per_table = 1

Innodb_flush_log_at_trx_commit = 2

Innodb_log_buffer_size = 64m

Innodb_buffer_pool_size = 4G

Innodb_thread_concurrency = 8

Innodb_flush_method = O_DIRECT

Innodb_log_file_size = 512m

[mysqld_safe]

Log-error=/var/log/mysqld.log

Pid-file=/var/run/mysqld/mysqld.pid

Sql_mode=STRICT_ALL_TABLES

Get the system initial password:

[root@hadoop01 wjg] # cat / root/.mysql_secret

# The random password set for the root user at Tue Dec 18 20:37:59 2018 (local time): pP4xR06U9nPyWrhA

Log in to MySQL:

[root@hadoop01 wjg] # mysql-uroot-ppP4xR06U9nPyWrhA

Warning: Using a password on the command line interface can be insecure.

Welcome to the MySQL monitor. Commands end with; or\ g.

Your MySQL connection id is 2

Server version: 5.6.42

SET PASSWORD FOR 'root'@'localhost' = PASSWORD (' xxxxxxx')

Grant all on. To root@ "" identified by "xxxxxxx"

Set MySQL Boot self-boot

Systemctl enable mysql.service

Initialize MySQL

/ usr/bin/mysql_secure_installation

Download mysql-connector-java-5.1.46.tar.gz and extract it

Find mysql-connector-java.jar in the unzipped directory and copy it to the / usr/share/java directory of all hosts

Cd mysql-connector-java-5.1.46

Cp mysql-connector-java-5.1.46.jar / usr/share/java/mysql-connector-java.jar

[root@hadoop01 usr] # ls-l / usr/share/java/mysql-connector-java.jar

-rw-r--r-- 1 root root 1004838 Dec 19 20:45 / usr/share/java/mysql-connector-java.jar

Download Oracle JDK package oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm and upload it to all hosts and install it

Download address: https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPMS/x86_64/

Rpm-ivh oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm

Creating Databases for Cloudera Software

The libraries that need to be built are

Scm 、 amon 、 rman 、 hue 、 metastore 、 sentry 、 nav 、 navms 、 oozie

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci

GRANT ALL ON scm. TO 'scm'@'%' IDENTIFIED BY' scm@123'

CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci

GRANT ALL ON rman. TO 'rman'@'%' IDENTIFIED BY' rman@123'

CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci

GRANT ALL ON hue. TO 'hue'@'%' IDENTIFIED BY' hue@123'

CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci

GRANT ALL ON metastore. TO 'metastore'@'%' IDENTIFIED BY' metastore@123'

CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci

GRANT ALL ON sentry. TO 'sentry'@'%' IDENTIFIED BY' sentry@123'

CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci

GRANT ALL ON nav. TO 'nav'@'%' IDENTIFIED BY' nav@123'

CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci

GRANT ALL ON navms. TO 'navms'@'%' IDENTIFIED BY' navms@123'

CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci

GRANT ALL ON oozie. TO 'oozie'@'%' IDENTIFIED BY' oozie@123'

Initialize the cloudera manager database

/ opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm

Download and install Cloudera Manager

Https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPMS/x86_64/

Cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpmcloudera-manager-server-6.0.1-610811.el7.x86_64.rpmcloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm

Install 1, 1, 2 and 3 on the hadoop01 server of the master node

Install 1Pol 3 on the Agent server

Execute on the primary node hadoop01:

Systemctl start cloudera-scm-server

Systemctl restart cloudera-scm-server

View the log:

Tail-f / var/log/cloudera-scm-server/cloudera-scm-server.log

The display below indicates that the service is normal.

INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

Cloudera Manager will automatically download files to the / opt/cloudera/parcel-repo directory

CDH-6.0.1-1.cdh7.0.1.p0.590678-el7.parcel

CDH-6.0.1-1.cdh7.0.1.p0.590678-el7.parcel.sha

CDH-6.0.1-1.cdh7.0.1.p0.590678-el7.parcel.torrent

Manifest.json

Execute on all Agent nodes hadoop01, hadoop02, hadoop06:

Configure the Server pointing to the Agent

Vi / etc/cloudera-scm-agent/config.ini

[General]

# Hostname of the CM server.

Server_host=hadoop01

[Security]

# Use TLS and certificate validation when connecting to the CM server.

Use_tls=1

Start the cloudera-scm-agent service:

Systemctl status cloudera-scm-agent

Systemctl start cloudera-scm-agent

Systemctl restart cloudera-scm-agent

Agent Log file location: / var/log/cloudera-scm-agent

Cloudera-scm-agent.log

Tail-f / var/log/cloudera-scm-agent/cloudera-scm-agent.log

Log in to the Cloudera Manager administration interface

Http://192.168.1.101:7180

Both user name and password are admin

Cloudera installation Parcel prompts the host to be in poor health

Solution method

Delete the cm_guid file under the agent directory and restart the agent service of the failed node.

[root@hadoop06 ~] # cd / var/lib/cloudera-scm-agent/

[root@hadoop06 cloudera-scm-agent] # ll

Total 32

-rw- 1 root root 2 Dec 28 14:27 active_parcels.json

Drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Dec 28 23:01 agent-cert

-rw-r--r-- 1 root root 36 Dec 28 14:25 cm_guid

-rw- 1 root root 13813 Dec 29 07:10 response.avro

-rw-r--r-- 1 root root 36 Dec 28 14:25 uuid

[root@hadoop06 cloudera-scm-agent] # rm-rf cm_guid

[root@hadoop06 cloudera-scm-agent] # systemctl restart cloudera-scm-agent

After you remove cm_guid on all nodes, cloudera manager automatically continues with the installation.

After a little while, you will see that parcel has been installed on all hosts and activated.

Click continue to enter the interface to check the correctness of the host.

Warning 1: Cloudera recommends setting / proc/sys/vm/swappiness to a maximum of 10. Currently set to 30. Use the sysctl command to change the setting at run time and edit / etc/sysctl.conf to save the setting after reboot. You can continue with the installation, but Cloudera Manager may report that your host is in poor health due to swapping. The following hosts will be affected:

View details

Hadoop [01-02, 06]

Solution:

Temporary adjustment: sysctl vm.swappiness=10

Permanent adjustment: vi / etc/sysctl.conf

Add the following line

Vm.swappiness=10

Second warning: transparent large page compression is enabled, which can cause significant performance problems. Run "echo never > / sys/kernel/mm/transparent_hugepage/defrag" and "echo never > / sys/kernel/mm/transparent_hugepage/enabled" to disable this setting, and then add the same command to initialization scripts such as / etc/rc.local to set it when the system is rebooted. The following hosts will be affected:

View details

Hadoop [01-02, 06]

Solution:

Echo never > / sys/kernel/mm/transparent_hugepage/defrag

Echo never > / sys/kernel/mm/transparent_hugepage/enabled

Modify / etc/rc.local to add the above two lines

Vi / etc/rc.local

Echo never > / sys/kernel/mm/transparent_hugepage/defrag

Echo never > / sys/kernel/mm/transparent_hugepage/enabled

Third warning: handling of psycopg2 version is too low

Install pip

Yum-y install epel-release

Yum-y install python-pip

Pip install-upgrade pip

Yum install postgresql-devel

Install psycopg2

Download psycopg2-2.7.6.1.tar.gz and extract it

[root@hadoop01 psycopg2-2.7.6.1] # python setup.py build

[root@hadoop01 psycopg2-2.7.6.1] # python setup.py install

Rerun the check host, and all project checks pass

Click to continue to go to the select and install product interface

Select all services and click continue to enter the host role assignment interface

Click by host to view the roles assigned to the host will see more clearly

Click continue to enter the database settings interface and fill in the previously created database name, user name and password

Click to continue to enter the review changes interface

Click to continue to start the installation

Click continue after installation to enter the Summary interface

At this point, the installation of Cloudera Manager6.0.1 has been completed successfully. Next, let's log in quickly and have a look.

Https://192.168.1.101:7183/cmf/home

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.