Network Security Internet Technology Development Database Servers Mobile Phone Android Software Apple Software Computer Software News IT Information

In addition to Weibo, there is also WeChat

Please pay attention

WeChat public account

Shulou

Build CDH5.8.3 under centos7.2

2025-01-16 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Database >

Share

Shulou(Shulou.com)06/01 Report--

1. Basic environment:

Operating system:

Centos 7.2.1511

Jdk environment

Version: jdk-8u45-linux-x64.rpm

Mysql environment:

Rpm package: http://ftp.ntu.edu.tw/MySQL/Downloads/MySQL-5.6/MySQL-5.6.33-1.linux_glibc2.5.x86_64.rpm-bundle.tar

Jdbc link package: http://101.96.10.72/cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.40.tar.gz

CDH installation related packages:

Cloudera manager package: http://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.8.3_x86_64.tar.gz

CDH package: http://archive.cloudera.com/cdh6/parcels/5.8.3/CDH-5.8.3-1.cdh6.8.3.p0.2-el7.parcel.sha1

Http://archive.cloudera.com/cdh6/parcels/5.8.3/CDH-5.8.3-1.cdh6.8.3.p0.2-el7.parcel

Http://archive.cloudera.com/cdh6/parcels/5.8.3/manifest.json

Cluster planning

IP address hostname description

192.168.50.123hadoop1 master node master,datanode

192.168.50.124hadoop2datanode

192.168.50.125hadoop3 datanode

Pre-installation configuration

1. Install jdk (on every machine)

Uninstall the original jdk version before installation to avoid conflicts

two。 Modify the hosts on three machines

192.168.50.123 hadoop1

192.168.50.124 hadoop2

192.168.50.125 hadoop3

3. Synchronization time

Ntpdate-s pool.ntp.org

4. Turn off the firewall and selinux

Sed-I's Charpy SELINUXUnix. Restart the machine by using SELINUXA / etc/selinux/config #

Systemctl stop firewalld

Systemctl disable firewalld

5. Configure ssh login without password

[root@localhost ~] # ssh-keygen-t rsa-P''Generating public/private rsa key pair.Enter file in which to save the key (/ root/.ssh/id_rsa): Your identification has been saved in / root/.ssh/id_rsa.Your public key has been saved in / root/.ssh/id_rsa.pub.The key fingerprint is:62:b0:4c:aa:e5:37:92:89:4d:db:c3:38:e2:f1:2a:d6 root@admin -nodeThe key's randomart p_w_picpath is:+-- [RSA 2048]-+ | o | | + o | | + o o S | | B B. . | | +. @ * | | oooE o | | oo.. | | | +-+

Ssh-copy-id hadoop1

Ssh-copy-id hadoop2

Ssh-copy-id hadoop3

6. Install mysql

[root@hadoop1] # tar-xvf MySQL-5.6.33-1.linux_glibc2.5.x86_64.rpm-bundle.tar [root@hadoop1] # rpm- ivh MySQL-*.rpm modify configuration file path: cp / usr/share/mysql/my-default.cnf / etc/my.cnf

# configure mysql

[root@hadoop1] # vim / etc/ my.cnf [mysqld] default-storage-engine = innodbinnodb_file_per_tablecollation-server = utf8_general_ciinit-connect = 'SET NAMES utf8'character-set-server = utf8

# initialize the database

/ usr/local/mysql/scripts/mysql_install_db-basedir=/usr/local/mysql/-datadir=/data/mysql/-user=mysql > > / dev/null

# start mysql

Service mysqld startchkconfig mysqld on

-View mysql root initialization password

[root@hadoop1] # cat / root/.mysql_secret# The random password set for the root user at Fri Sep 16 11:13:25 2016 (local time): 9mp7uYFmgt6drdq3-Log in to change the password [root@hadoop1] # mysql-u root-pmysql > SET PASSWORD=PASSWORD ('123456');-allow mysql remote access to mysql > grant all on *. * to root@ "%" Identified by "www.123"; Query OK, 1 row affected (0.05 sec) mysql > flush privileges;Query OK, 0 rows affected (0.00 sec)

Create the libraries required for cdh

Create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;Query OK, 1 row affected (0.00 sec) create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;Query OK, 1 row affected (0.00 sec) create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;Query OK, 1 row affected (0.00 sec) create database monitor DEFAULT CHARSET utf8 COLLATE utf8_general_ci;Query OK, 1 row affected (0.00 sec) create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;Query OK, 1 row affected (0.00 sec)

7. Third-party dependent package installation (all nodes are installed)

Yum install chkconfig python bind-utils psmisc libxslt zlib sqlite fuse fuse-libs redhat-lsb cyrus-sasl-plain cyrus-sasl-gssapi

Note that this place depends on the package does not install the following startup cluster will not be able to start, this is a bloody lesson ah!

Prepare mysql's jar package on hadoop1

[root@hadoop1] # mkdir-p / usr/share/java modify the name of the jar package and copy it to the / usr/share/java/ directory [root@hadoop1] # cp mysql-connector-java-5.1.40-bin.jar / usr/share/java/mysql-connector-java.jar

8. Install Cloudera-Manager

Extract the cm package to the specified directory, and all servers need to do it.

[root@hadoop1 ~] # mkdir / opt/cloudera-manager [root@hadoop1 ~] # tar-axvf cloudera-manager-centos7-cm5.8.3_x86_64.tar.gz-C / opt/cloudera-manager

Create cloudera-scm users (all nodes)

[root@hadoop1] # useradd-r-d / opt/cloudera-manager/cm-5.8.3/run/cloudera-scm-server-M-c "Cloudera SCM User" cloudera-scm

Configure agent in hadoop2 and hadoop3

Vim / opt/cloudera-manager/cm-5.8.3/etc/cloudera-scm-agent/config.ini change server_host to the hostname where CMS is located, namely hadoop1

Create a parcel-repo repository in the master node

[root@hadoop1 ~] # chown cloudera-scm:cloudera-scm / opt/cloudera/parcel-repo [root@hadoop1 ~] # mv CDH-5.8.3- 1.cdh6.8.3.p0.2-el7.parcel.sha1 CDH-5.8.3- 1.cdh6.8.3.p0.2-el7.parcel.sha [root@hadoop1 ~] # cp CDH-5.8.3- 1.cdh6.8.3.p0.2-el7.parcel CDH-5.8.3- 1.cdh6.8.3.p0.2-el7.parcel.sha manifest.json / opt/cloudera/parcel-repo

Explanation: Clouder-Manager extracts CDHs from the / opt/cloudera/parcel-repo directory of the master node, distributes and decompresses it and activates it into the / opt/cloudera/parcels directory of each node

Initial script configuration database scm_prepare_database.sh (on the primary node)

[root@hadoop1] # / opt/cloudera-manager/cm-5.8.3/share/cmf/schema/scm_prepare_database.sh mysql-h hadoop1-P 3306-uroot-pwww.123-- scm-host master scm scm scmJAVA_HOME=/usr/java/jdk1.8.0_45Verifying that we can write to / opt/cloudera-manager/cm-5.8.3/etc/cloudera-scm-serverCreating SCM configuration file in / opt/cloudera-manager/cm-5.8.3/etc/ Cloudera-scm-serverExecuting: / usr/java/jdk1.8.0_45/bin/java-cp / usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera-manager/cm-5.8.3/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor / opt/cloudera-manager/cm-5.8.3/etc/cloudera-scm-server/db.properties com.cloudera .cmf.db. [main] DbCommandExecutor INFO Successfully connected to database.All done Your SCM database is configured correctly!

Description: this script is used to create and configure the database required by CMS. The parameters refer to:

Mysql: the database uses mysql, so if you use oracle during installation, this parameter should be changed to oracle.

-hhadoop1: the database is built on the hadoop1 host. That is, above the primary node.

-run mysql as uroot:root. -the root password for 123456:mysql is *.

The host of scm-host hadoop1:CMS is usually on the same host as the host installed by mysql.

The last three parameters are: database name, database user name, database password.

10. Start the agent service on each node

/ opt/cloudera-manager/cm-5.8.3/etc/init.d/cloudera-scm-agent start

Start the server service in master

/ opt/cloudera-manager/cm-5.8.3/etc/init.d/cloudera-scm-server start

Browser access

Http://192.168.50.123:7180/cmf/login username admin password admin

Question 1:

Service cloudera-scm-server status

Cloudera-scm-server dead but pid file exists

Solve

[root@master cm-5.8.3] # rm / root/hadoop/cm-5.8.3/run/cloudera-scm-server.pid

[root@master hadoop] #. / cm-5.8.3/etc/init.d/cloudera-scm-server restart

Cloudera-scm-server is already stopped

Starting cloudera-scm-server: [OK]

Question 2:

2016-12-08 03 ERROR WebServerImpl:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager 40 doesn't exist 57 479: The server storage directory [/ var/lib/cloudera-scm-server]

2016-12-08 03 var/lib/cloudera-scm-server 40 ERROR WebServerImpl:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: No read permission to the server storage directory [/ var/lib/cloudera-scm-server]

2016-12-08 03 var/lib/cloudera-scm-server 40 ERROR WebServerImpl:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: No write permission to the server storage directory [/ var/lib/cloudera-scm-server]

Resolve:

Successful after creating the directory and adding permissions

Mkdir / var/lib/cloudera-scm-server

Chown-R cloudera-scm.cloudera-scm / var/lib/cloudera-scm-server

Question 3: check where there are two warnings on the host in CDH

Resolve:

Echo never > / sys/kernel/mm/transparent_hugepage/defrag

Echo 10 > / proc/sys/vm/swappiness

Reference website:

Http://blog.csdn.net/shawnhu007/article/details/52579204

Http://www.cnblogs.com/itboys/p/5955545.html

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.

Views: 0

*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.

Share To

Database

Wechat

© 2024 shulou.com SLNews company. All rights reserved.

12
Report