In addition to Weibo, there is also WeChat
Please pay attention
WeChat public account
Shulou
2025-01-16 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Database >
Share
Shulou(Shulou.com)06/01 Report--
1. Basic environment:
Operating system:
Centos 7.2.1511
Jdk environment
Version: jdk-8u45-linux-x64.rpm
Mysql environment:
Rpm package: http://ftp.ntu.edu.tw/MySQL/Downloads/MySQL-5.6/MySQL-5.6.33-1.linux_glibc2.5.x86_64.rpm-bundle.tar
Jdbc link package: http://101.96.10.72/cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.40.tar.gz
CDH installation related packages:
Cloudera manager package: http://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.8.3_x86_64.tar.gz
CDH package: http://archive.cloudera.com/cdh6/parcels/5.8.3/CDH-5.8.3-1.cdh6.8.3.p0.2-el7.parcel.sha1
Http://archive.cloudera.com/cdh6/parcels/5.8.3/CDH-5.8.3-1.cdh6.8.3.p0.2-el7.parcel
Http://archive.cloudera.com/cdh6/parcels/5.8.3/manifest.json
Cluster planning
IP address hostname description
192.168.50.123hadoop1 master node master,datanode
192.168.50.124hadoop2datanode
192.168.50.125hadoop3 datanode
Pre-installation configuration
1. Install jdk (on every machine)
Uninstall the original jdk version before installation to avoid conflicts
two。 Modify the hosts on three machines
192.168.50.123 hadoop1
192.168.50.124 hadoop2
192.168.50.125 hadoop3
3. Synchronization time
Ntpdate-s pool.ntp.org
4. Turn off the firewall and selinux
Sed-I's Charpy SELINUXUnix. Restart the machine by using SELINUXA / etc/selinux/config #
Systemctl stop firewalld
Systemctl disable firewalld
5. Configure ssh login without password
[root@localhost ~] # ssh-keygen-t rsa-P''Generating public/private rsa key pair.Enter file in which to save the key (/ root/.ssh/id_rsa): Your identification has been saved in / root/.ssh/id_rsa.Your public key has been saved in / root/.ssh/id_rsa.pub.The key fingerprint is:62:b0:4c:aa:e5:37:92:89:4d:db:c3:38:e2:f1:2a:d6 root@admin -nodeThe key's randomart p_w_picpath is:+-- [RSA 2048]-+ | o | | + o | | + o o S | | B B. . | | +. @ * | | oooE o | | oo.. | | | +-+
Ssh-copy-id hadoop1
Ssh-copy-id hadoop2
Ssh-copy-id hadoop3
6. Install mysql
[root@hadoop1] # tar-xvf MySQL-5.6.33-1.linux_glibc2.5.x86_64.rpm-bundle.tar [root@hadoop1] # rpm- ivh MySQL-*.rpm modify configuration file path: cp / usr/share/mysql/my-default.cnf / etc/my.cnf
# configure mysql
[root@hadoop1] # vim / etc/ my.cnf [mysqld] default-storage-engine = innodbinnodb_file_per_tablecollation-server = utf8_general_ciinit-connect = 'SET NAMES utf8'character-set-server = utf8
# initialize the database
/ usr/local/mysql/scripts/mysql_install_db-basedir=/usr/local/mysql/-datadir=/data/mysql/-user=mysql > > / dev/null
# start mysql
Service mysqld startchkconfig mysqld on
-View mysql root initialization password
[root@hadoop1] # cat / root/.mysql_secret# The random password set for the root user at Fri Sep 16 11:13:25 2016 (local time): 9mp7uYFmgt6drdq3-Log in to change the password [root@hadoop1] # mysql-u root-pmysql > SET PASSWORD=PASSWORD ('123456');-allow mysql remote access to mysql > grant all on *. * to root@ "%" Identified by "www.123"; Query OK, 1 row affected (0.05 sec) mysql > flush privileges;Query OK, 0 rows affected (0.00 sec)
Create the libraries required for cdh
Create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;Query OK, 1 row affected (0.00 sec) create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;Query OK, 1 row affected (0.00 sec) create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;Query OK, 1 row affected (0.00 sec) create database monitor DEFAULT CHARSET utf8 COLLATE utf8_general_ci;Query OK, 1 row affected (0.00 sec) create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;Query OK, 1 row affected (0.00 sec)
7. Third-party dependent package installation (all nodes are installed)
Yum install chkconfig python bind-utils psmisc libxslt zlib sqlite fuse fuse-libs redhat-lsb cyrus-sasl-plain cyrus-sasl-gssapi
Note that this place depends on the package does not install the following startup cluster will not be able to start, this is a bloody lesson ah!
Prepare mysql's jar package on hadoop1
[root@hadoop1] # mkdir-p / usr/share/java modify the name of the jar package and copy it to the / usr/share/java/ directory [root@hadoop1] # cp mysql-connector-java-5.1.40-bin.jar / usr/share/java/mysql-connector-java.jar
8. Install Cloudera-Manager
Extract the cm package to the specified directory, and all servers need to do it.
[root@hadoop1 ~] # mkdir / opt/cloudera-manager [root@hadoop1 ~] # tar-axvf cloudera-manager-centos7-cm5.8.3_x86_64.tar.gz-C / opt/cloudera-manager
Create cloudera-scm users (all nodes)
[root@hadoop1] # useradd-r-d / opt/cloudera-manager/cm-5.8.3/run/cloudera-scm-server-M-c "Cloudera SCM User" cloudera-scm
Configure agent in hadoop2 and hadoop3
Vim / opt/cloudera-manager/cm-5.8.3/etc/cloudera-scm-agent/config.ini change server_host to the hostname where CMS is located, namely hadoop1
Create a parcel-repo repository in the master node
[root@hadoop1 ~] # chown cloudera-scm:cloudera-scm / opt/cloudera/parcel-repo [root@hadoop1 ~] # mv CDH-5.8.3- 1.cdh6.8.3.p0.2-el7.parcel.sha1 CDH-5.8.3- 1.cdh6.8.3.p0.2-el7.parcel.sha [root@hadoop1 ~] # cp CDH-5.8.3- 1.cdh6.8.3.p0.2-el7.parcel CDH-5.8.3- 1.cdh6.8.3.p0.2-el7.parcel.sha manifest.json / opt/cloudera/parcel-repo
Explanation: Clouder-Manager extracts CDHs from the / opt/cloudera/parcel-repo directory of the master node, distributes and decompresses it and activates it into the / opt/cloudera/parcels directory of each node
Initial script configuration database scm_prepare_database.sh (on the primary node)
[root@hadoop1] # / opt/cloudera-manager/cm-5.8.3/share/cmf/schema/scm_prepare_database.sh mysql-h hadoop1-P 3306-uroot-pwww.123-- scm-host master scm scm scmJAVA_HOME=/usr/java/jdk1.8.0_45Verifying that we can write to / opt/cloudera-manager/cm-5.8.3/etc/cloudera-scm-serverCreating SCM configuration file in / opt/cloudera-manager/cm-5.8.3/etc/ Cloudera-scm-serverExecuting: / usr/java/jdk1.8.0_45/bin/java-cp / usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera-manager/cm-5.8.3/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor / opt/cloudera-manager/cm-5.8.3/etc/cloudera-scm-server/db.properties com.cloudera .cmf.db. [main] DbCommandExecutor INFO Successfully connected to database.All done Your SCM database is configured correctly!
Description: this script is used to create and configure the database required by CMS. The parameters refer to:
Mysql: the database uses mysql, so if you use oracle during installation, this parameter should be changed to oracle.
-hhadoop1: the database is built on the hadoop1 host. That is, above the primary node.
-run mysql as uroot:root. -the root password for 123456:mysql is *.
The host of scm-host hadoop1:CMS is usually on the same host as the host installed by mysql.
The last three parameters are: database name, database user name, database password.
10. Start the agent service on each node
/ opt/cloudera-manager/cm-5.8.3/etc/init.d/cloudera-scm-agent start
Start the server service in master
/ opt/cloudera-manager/cm-5.8.3/etc/init.d/cloudera-scm-server start
Browser access
Http://192.168.50.123:7180/cmf/login username admin password admin
Question 1:
Service cloudera-scm-server status
Cloudera-scm-server dead but pid file exists
Solve
[root@master cm-5.8.3] # rm / root/hadoop/cm-5.8.3/run/cloudera-scm-server.pid
[root@master hadoop] #. / cm-5.8.3/etc/init.d/cloudera-scm-server restart
Cloudera-scm-server is already stopped
Starting cloudera-scm-server: [OK]
Question 2:
2016-12-08 03 ERROR WebServerImpl:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager 40 doesn't exist 57 479: The server storage directory [/ var/lib/cloudera-scm-server]
2016-12-08 03 var/lib/cloudera-scm-server 40 ERROR WebServerImpl:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: No read permission to the server storage directory [/ var/lib/cloudera-scm-server]
2016-12-08 03 var/lib/cloudera-scm-server 40 ERROR WebServerImpl:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: No write permission to the server storage directory [/ var/lib/cloudera-scm-server]
Resolve:
Successful after creating the directory and adding permissions
Mkdir / var/lib/cloudera-scm-server
Chown-R cloudera-scm.cloudera-scm / var/lib/cloudera-scm-server
Question 3: check where there are two warnings on the host in CDH
Resolve:
Echo never > / sys/kernel/mm/transparent_hugepage/defrag
Echo 10 > / proc/sys/vm/swappiness
Reference website:
Http://blog.csdn.net/shawnhu007/article/details/52579204
Http://www.cnblogs.com/itboys/p/5955545.html
Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.
Views: 0
*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.
Continue with the installation of the previous hadoop.First, install zookooper1. Decompress zookoope
"Every 5-10 years, there's a rare product, a really special, very unusual product that's the most un
© 2024 shulou.com SLNews company. All rights reserved.