Shulou(Shulou.com)06/01 Report--

In this article, the editor introduces in detail "how to build and deploy the CDH environment". The content is detailed, the steps are clear, and the details are handled properly. I hope this article "how to build and deploy the CDH environment" can help you solve your doubts.

I. Overview

Cloudera version (Cloudera's Distribution Including Apache Hadoop, referred to as "CDH"), a Web-based user interface that supports most Hadoop components, including HDFS, MapReduce, Hive, Pig, Hbase, Zookeeper, Sqoop.

II. Installation and deployment 2.1 Cluster Architecture serial number IP address hostname system version 110.211.55.100zeroCentos 7.7210.211.55.101Hadoop1Centos 7.7310.211.55.102Hadoop2Centos 7.72.2 Base Environment configuration 2.2.1 Network configuration vi / etc/sysconfig/network-scripts/ifcfg-eth0BOOTPROTO=static ONBOOT=yesIPADDR=10.211.55.100NETMASK=255.255.255.0 GATEWAY=10.211.55.1systemctl restart network.service # restart Network Services 2.2.2 disable Firewall and SELINUXsystemctl stop firewalldsystemctl disable firewalldvi / etc/sysconfig/selinuxSELINUX=disabled # change the value of SELINUX to disabledreboot # restart the device to make the configuration of selinux take effect 2.2.3 change the hostname hostnamectl set-hostname zero2.2.3 modify hostsvi / etc/hosts10.211.55.100 zero10.211.55.101 hadoop110.211.55.102 hadoop2reboot # restart after modification 2.2.4 synchronization time yum-y install ntp # install NPT service vi / etc/ntp .conf # modify configuration file # comment out the original server pointing to # server 0.centos.pool.ntp.org iburst#server 1.centos.pool.ntp.org iburst#server 2.centos.pool.ntp.org iburst#server 3.centos.pool.ntp.org iburst#zero node pointing to server 0.cn.pool.ntp.orgserver 0.asia.pool.ntp.orgserver 3.asia.pool.ntp.orgserver 10.211.55.100 iburst# Hadoop1/Hadoop2 node pointing to zero Node ntpdate-u cn.pool.ntp.org # synchronize network time # restart ntp service systemctl start ntpdsystemctl enable ntpdhwclock-- localtime-w # write system time to local hardware clock time 2.2.5 configure SSH secret-free login ssh-keygen-t rsa # do not enter password Enter ssh-copy-id hadoop1ssh-copy-id hadoop2# all the way to test connectivity ssh 'zero'ssh' hadoop1'ssh 'hadoop2'2.2.6 install JDKmkdir / usr/Java/ # create directory tar-zxvf jdk1.8.0_211.tar.gz-C / usr/Java/ # extract the JDK package to the specified directory vi / etc/profile # configure environment variable # JAVAexport JAVA_HOME=/usr/Java/jdk1.8.0_211export JRE_HOME=$ {JAVA _ HOME} / jre export CLASSPATH=.:$ {JAVA_HOME} / lib:$ {JRE_HOME} / lib export PATH=$ {JAVA_HOME} / bin:$PATHsource / etc/profile # to make the configuration file effective java-version # check the version information to check whether the installation was successful # the return information after successful installation java version "1.8.0U211" Java (TM) SE Runtime Environment (build 1.8.0_211-b12) Java HotSpot (TM) 64-Bit Server VM (build 25.211-b12 Mixed mode) 2.2.7 install MySQL (just install the master node) # check whether the MySQL service is installed in the system. If it exists, delete MySQL and its dependent package rpm-qa | grep mysqlyum list installed | grep mysqlwget http://repo.mysql.com/mysql57-community-release-el7-8.noarch.rpmrpm-ivh mysql57-community-release-el7-8.noarch.rpmyum install mysql-server # install MySQL if prompted all the way to the end.

After installation, run MySQL, and a random password is automatically generated in the / var/log/mysqld.log file.

Service mysqld start # run MySQLgrep "password" / var/log/mysqld.log # to get a random password for logging in to the MySQL server # the returned information is as follows: "6WjiI.UxuoaF" is the MySQL login password 2021-01-05T11:04:28.684402Z 1 [Note] A temporary password is generated for root@localhost: 6WjiI.UxuoaF

Log in to the MySQL server and change the password.

/ * change password * / SET PASSWORD = PASSWORD ('new password'); ALTER USER' root'@'localhost' PASSWORD EXPIRE NEVER;flush privileges;/* set user root to be accessed under any IP * / grant all privileges on *. * to root@ "%" identified by "new password"; / * set user root to be accessed locally * / grant all privileges on *. * to root@ "localhost" identified by "new password"; / * refresh permissions to make it effective * / flush privileges 2.3 installation and deployment of Cloudera Manager 2.3.1 install Cloudera Manageryum-y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs redhat-lsb # install third-party dependency package mkdir-p / opt/cloudera-manager # create directory

Extract the Cloudera Manager package to the / opt/cloudera-manager directory.

Tar-zvxf cloudera-manager-centos7-cm5.14.0_x86_64.tar.gz-C / opt/cloudera-manager/2.3.2 configuration Cloudera Manager

Modify the agent configuration file.

The vi / opt/cloudera-manager/cm-5.14.0/etc/cloudera-scm-agent/config.ini# configuration file is modified as follows: # Hostname of the CM server.server_host=zero# Port that the CM server is listening on.server_port=7182

Create a dedicated normal user cloudera-scm for CM.

Useradd-- system # indicates that the system user is created-- home=/opt/cloudera-manager/cm-5.14.0/run/cloudera-scm-server # specifies the user's home directory-- no-create-home # indicates that the user's home directory is no longer created-- shell=/bin/false # is not used as a login user-- comment "Cloudera SCM User" cloudera-scm # user name

Configure database access and add users.

Grant all privileges on *. * to 'temp'@'%' identified by' password' with grant option;flush privileges

Execute the initialization script for CM in MySQL on the primary node.

Mv mysql-connector-java.jar / usr/share/java/mysql-connector-java.jar# enter the initialization steps of cd / opt/cloudera-manager/cm-5.14.0/share/cmf/schema# under the initialization script directory. / scm_prepare_database.sh mysql- h zero-utemp-ppassword-- scm-host zero scm scm scm

Configure CDH source (note that version is needed when downloading, and el5/6/7 corresponds to CentOS5/6/7 respectively. )

# create a directory mkdir-p / opt/cloudera/parcel-repo # master node mkdir-p / opt/cloudera/parcels # all nodes # move the parcel file to the parcel-repo directory mv CDH-5.14.0-1.cdh6.14.0.p0.24-el7.parcel / opt/cloudera/parcel-repo/# move the json file to the parcel-repo directory mv manifest.json / opt/cloudera/parcel-repo/# rename check text And move to mv CDH-5.14.0-1.cdh6.14.0.p0.24-el7.parcel.sha1 / opt/cloudera/parcel-repo/CDH-5.14.0-1.cdh6.14.0.p0.24-el7.parcel.sha under the parcel-repo directory

Change catalog owner

Chown-R cloudera-scm:cloudera-scm / opt/cloudera/parcel-repo/ # Master chown-R cloudera-scm:cloudera-scm / opt/cloudera/parcels/ # all nodes 2.3.3 start the CM process service

Start server (primary node)

/ opt/cloudera-manager/cm-5.14.0/etc/init.d/cloudera-scm-server startStarting cloudera-scm-server: [OK] # return information after successful startup

Start agent (all nodes)

/ opt/cloudera-manager/cm-5.14.0/etc/init.d/cloudera-scm-agent startStarting cloudera-scm-agent: [OK] # return message after successful startup 2.3.3 start deployment

Open http://10.211.55.100:7180/ to enter the CM initial interface, and the account password is admin.

Log in and check the agreed terms, click continue, and select the free version.

To continue to the next step, check to install the host.

Cluster installation can be checked by default, click continue and wait for the installation to complete.

Click continue to check the correctness of the host and repair it with reference to the information given.

Click continue to select the desired service.

Click continue to assign roles.

Configure the database to connect successfully.

Review the changes page directly by default, click continue.

Wait for the deployment to complete.

Click continue to complete the installation.

Error & solve # error Your password does not satisfy the current policy requirements# solution vi / etc/my.cnf # modify configuration file my.cnfvalidate_password=off # close password policy setting validate_passwordservice mysqld restart # restart service # error specified host for CDH cluster installation cannot be checked # solution / opt/cloudera-manager/cm-5.14.0/etc/init.d/cloudera-scm-server stop # stop server/opt/cloudera-manager/cm-5.14.0/etc/init.d/cloudera-scm-agent stop # stop agent# delete the content in the / opt/cloudera-manager/cm-5.14.0/lib/cloudera-scm-server-db/data/ directory rm-rf / opt/cloudera-manager/cm-5.14. Delete the data in the hosts table under the scm database in the MySQL database * / use scm Delete from HOSTS # error MainThread agent ERROR Error, CM server guid updated, expected 7cda865f-a833-40d1-9fa6-761631c74326 Received e2678b78-7c5e-4703-91b6-c52e2875988c# solution rm-rf / opt/cloudera-manager/cm-5.14.0/lib/cloudera-scm-agent/cm_guid# error ERROR Table 'scm.CM_VERSION' doesn't exist# solution vi / etc/my.cnf # modify configuration file my.cnflower_case_table_names = 1 # [mysqld] node add service mysqld restart # restart service # error ERROR WebServerImpl:com.cloudera. Server.web.cmf.search.components.SearchRepositoryManager: The server storage directory [/ var/lib/cloudera-scm-server] doesn't exist.# solution mkdir / var/lib/cloudera-scm-serverchown-R cloudera-scm:cloudera-scm / var/lib/cloudera-scm-server # create a directory and add permissions to read here This article "how to build and deploy the CDH environment" has been introduced. If you want to master the knowledge points of this article, you still need to practice and use it. If you want to know more about the article, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.