Shulou(Shulou.com)06/01 Report--

This article mainly shows you "how to install CDH5.16.1 in Redhat7.4", the content is easy to understand, clear, hope to help you solve your doubts, the following let the editor lead you to study and learn "how to install CDH5.16.1 in Redhat7.4" this article.

Pre-preparation 2.1.hostname and hosts configuration

Nodes in the cluster can communicate with each other using static IP addresses. The IP address and hostname are configured through / etc/hosts, and the hostname is configured through / etc/hostname.

Take the cm node (172.31.13.38) as an example:

Hostname configuration

/ etc/hostname file is as follows:

Ip-172-31-13-38.ap-southeast-1.compute.internal

Or you can modify it by order to take effect immediately.

[root@ip-172-31-13-38] $hostnamectl set-hostname ip-172-31-13-38.ap-southeast-1.compute.internal

Note: modify the difference between hostname and REDHAT6 here

Hosts configuration

/ etc/hosts file is as follows:

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain6172.31.13.38 ip-172-31-13-38.ap-southeast-1.compute.internal172.31.11.232 ip-172-31-11-232.ap-southeast-1.compute.internal172.31.11.9 ip-172-31-11-9.ap-southeast-1.compute.internal172.31.13.166 ip-172-31-13-166.ap-southeast-1.compute.internal

For the above two steps, configure other nodes in the cluster accordingly. Confirm the hosts files for the 4 hosts that need to be installed:

2.2. Disable SELinux

Execute the setenforce 0 command on all nodes, using batch shell here:

[root@ip-172-31-13-38] $sh ssh_do_all.sh node.list "setenforce 0"

All nodes in the cluster modify the / etc/selinux/config file as follows:

SELINUX=disabledSELINUXTYPE=targeted

Check that all machines have been modified successfully:

Sh ssh_do_all.sh node.list "cat / etc/selinux/config | grep SELINUX"

2.3. Turn off the firewall

All nodes in the cluster execute systemctl stop firewalld commands. Here, batch commands are executed through shell:

[root@ip-172-31-13-38] $sh ssh_do_all.sh node.list "systemctl stop firewalld" [root@ip-172-31-13-38] $sh ssh_do_all.sh node.list "systemctl disable firewalld" [root@ip-172-31-13-38 ~] $sh ssh_do_all.sh node.list "systemctl status firewalld"

2.4. Cluster clock synchronization

On Redhat7.x 's operating system, chrony has been installed by default, so let's uninstall chrony first and then install ntp. Use ntp to configure clock synchronization on each machine, using the cm (172.31.13.38) service as the local ntp server, with the other three servers synchronizing with it.

1. Uninstall chrony on all machines

[root@ip-172-31-13-38 shell] # sh ssh_do_all.sh node.list "yum-y remove chrony"

two。 All machines install ntp

[root@ip-172-31-13-38 shell] # sh ssh_do_all.sh node.list "yum-y install ntp"

3.cm machines configure clocks to synchronize with themselves

[root@ip-172-31-16-68 shell] # vim / etc/ntp.conf#server 0.rhel.pool.ntp.org iburst#server 1.rhel.pool.ntp.org iburst#server 2.rhel.pool.ntp.org iburst#server 3.rhel.pool.ntp.org iburstserver 127.127.1.0 # local clockfudge 127.127.1.0 stratum 10

4. Other nodes in the cluster are configured to find cm machines to synchronize.

# server 0.rhel.pool.ntp.org iburst#server 1.rhel.pool.ntp.org iburst#server 2.rhel.pool.ntp.org iburst#server 3.rhel.pool.ntp.org iburstserver 172.31.13.38

5. Restart the ntp service for all machines

[root@ip-172-31-13-38 shell] # sh ssh_do_all.sh node.list "systemctl restart ntpd" [root@ip-172-31-13-38 shell] # sh ssh_do_all.sh node.list "systemctl status ntpd" [root@ip-172-31-13-38 shell] # sh ssh_do_all.sh node.list "systemctl enable ntpd"

6. Verify that it is always synchronized and execute the ntpq-p command on all nodes, as shown in batch execution using scripts

[root@ip-172-31-13-38 shell] # sh ssh_do_all.sh node.list "ntpq-p"

The * sign on the left indicates that the synchronization is successful.

2.5. Set up swap

All nodes execute

[root@ip-172-31-13-38 shell] # sh ssh_do_all.sh node.list "echo vm.swappiness=10 > > / etc/sysctl.conf" [root@ip-172-31-13-38 shell] # sh ssh_do_all.sh node.list "sysctl vm.swappiness=10"

2.6. Set up large transparent pages

All nodes execute:

[root@ip-172-31-13-38 shell] # sh ssh_do_all.sh node.list "echo never > / sys/kernel/mm/transparent_hugepage/defrag" [root@ip-172-31-13-38 shell] # sh ssh_do_all.sh node.list "echo never > / sys/kernel/mm/transparent_hugepage/enabled"

Set the power on self-shutdown

Add the following script to the / etc/rc.d/rc.local file

If test-f / sys/kernel/mm/transparent_hugepage/enabled; then echo never > / sys/kernel/mm/transparent_hugepage/enabled fi if test-f / sys/kernel/mm/transparent_hugepage/defrag; then echo never > / sys/kernel/mm/transparent_hugepage/defrag fi

Synchronize to all nodes

2.7. Configure the operating system repo

Fayson uses the environment of AWS, which can be omitted and is put here for reference by the brothers who deploy the physical machine.

Mount the operating system iso file

[root@ip-172-31-2-159] $sudo mkdir / media/ DVD1 [root @ ip-172-31-2-159] $sudo mount-o loop CentOS-7-x86_64-DVD-1611.iso / media/DVD1/

Configure the operating system repo

[root@ip-172-31-2-15915] $sudo vim / etc/yum.repos.d/local_ os.repo [local _ iso] name=CentOS-$releasever-Mediabaseurl= file:///media/DVD1gpgcheck=0enabled=1[root@ip-172-31-2-159 ~] $sudo yum repolist2.8. Install the http service

Install the httpd service

[root@ip-172-31-2-159] $yum-y install httpd

Start or stop the httpd service

[root@ip-172-31-2-159d] $systemctl start httpd [root @ ip-172-31-2-159d] $systemctl stop httpd

After installing httpd, remake the operating system repo and change it to http so that other servers can also access it.

[root@ip-172-31-2-159] $sudo mkdir / var/www/html/ iso[ root @ ip-172-31-2-159] $sudo scp-r / media/DVD1/* / var/www/html/iso/ [root @ ip-172-31-2-159] $sudo vim / etc/yum.repos.d/os.repo [osrepo] name=os_repobaseurl= http://172.31.2.159/iso/enabled=truegpgcheck=false[root@ip-172-31-2-159] $sudo yum repolist2.9. Install MariaDB

1. Install MariaDB

[root@ip-172-31-13-38] # yum-y install MariaDB [root @ ip-172-31-13-38 ~] # yum-y install mariadb-server

two。 Start and configure MariaDB

[root@ip-172-31-13-38] # systemctl start MariaDB [root @ ip-172-31-13-38] # systemctl enable MariaDB [root @ ip-172-31-13-38] # / usr/bin/mysql_secure_installationNOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!In order to log into MariaDB to secure it, we'll need the currentpassword for the root user. If you've just installed MariaDB, andyou haven't set the root password yet, the password will be blank,so you should just press enter here.Enter current password for root (enter for none): OK, successfully used password, moving on...Setting the root password ensures that nobody can log into the MariaDBroot user without the proper authorisation.Set root password? [Y/n] YNew password: Re-enter new password: Password updated successfully!Reloading privilege tables.. ... Success!By default, a MariaDB installation has an anonymous user, allowing anyoneto log into MariaDB without having to have a user account created forthem. This is intended only for testing, and to make the installationgo a bit smoother. You should remove them before moving into aproduction environment.Remove anonymous users? [Y/n] Y... Successful normally, root should only be allowed to connect from 'localhost'. Thisensures that someone cannot guess at the root password from the network.Disallow root login remotely? [Y/n] n... Skipping.By default, MariaDB comes with a database named 'test' that anyone canaccess. This is also intended only for testing, and should be removedbefore moving into a production environment.Remove test database and access to it? [YPo] Y-Dropping test database... ... Success!-Removing privileges on test database... ... Success!Reloading the privilege tables will ensure that all changes made so farwill take effect immediately.Reload privilege tables now? [Y/n] Y... Success!Cleaning up...All done! If you've completed all of the above steps, your MariaDBinstallation should now be secure.Thanks for using MariaDB!

3. Set up the tables needed by CM,Hive, etc.

[root@ip-172-31-13-38] # mysql-u root-pEnter password: Welcome to the MariaDB monitor. Commands end with; or\ g.Your MariaDB connection id is 9Server version: 5.5.56-MariaDB MariaDB ServerCopyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.Type 'help;' or'\ h' for help. Type'\ c'to clear the current input statement.MariaDB [(none)] > create database metastore default character set utf8; CREATE USER 'hive'@'%' IDENTIFIED BY' password'; GRANT ALL PRIVILEGES ON metastore. * TO 'hive'@'%'; FLUSH PRIVILEGES; create database cm default character set utf8; CREATE USER' cm'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON cm. * TO 'cm'@'%'; FLUSH PRIVILEGES; create database am default character set utf8; CREATE USER' am'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON am. * TO 'am'@'%'; FLUSH PRIVILEGES; create database rm default character set utf8; CREATE USER' rm'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON rm. * TO 'rm'@'%'; FLUSH PRIVILEGES;create database hue default character set utf8; CREATE USER' hue'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON hue. * TO 'hue'@'%'; FLUSH PRIVILEGES;create database oozie default character set utf8; CREATE USER' oozie'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON oozie. * TO 'oozie'@'%'; FLUSH PRIVILEGES;create database sentry default character set utf8; CREATE USER' sentry'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON sentry. * TO 'sentry'@'%'; FLUSH PRIVILEGES;create database nav_ms default character set utf8; CREATE USER' nav_ms'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON nav_ms. * TO 'nav_ms'@'%'; FLUSH PRIVILEGES;create database nav_as default character set utf8; CREATE USER' nav_as'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON nav_as. * TO 'nav_as'@'%'; FLUSH PRIVILEGES

Install the jdbc driver

[root-31-13-38 ~] # mkdir-p / usr/share/java/ [root @ ip-172-31-13-38 ~] # mv mysql-connector-java-5.1.34.jar / usr/share/java/ [root @ ip-172-31-13-38 ~] # cd / usr/share/ Java [root @ ip-172-31-13-38 java] # ln-s mysql-connector-java-5.1.34.jar mysql-connector-java. Jar [root @ ip-172-31-13-38 java] # lltotal 940-rwxrwxr-x. 1 root root 960372 Feb 1 08:31 mysql-connector-java-5.1.34.jarlrwxrwxrwx 1 root root 31 Feb 2 00:52 mysql-connector-java.jar-> mysql-connector-java-5.1.34.jar

3.Cloudera Manager installation 3. 1. Configure a local repo source

1. Download the installation package for CM5.16.1 at:

Http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/cloudera-manager-agent-5.16.1-1.cm5161.p0.1.el7.x86_64.rpmhttp://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64. Rpm http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64.rpmhttp://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7. X86_64.rpm http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/enterprise-debuginfo-5.16.1-1.cm5161.p0.1.el7.x86_64.rpmhttp://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/jdk-6u31-linux-amd64.rpmhttp://archive.cloudera.com/cm5/redhat/7/ X86_64/cm/5.16.1/RPMS/x86_64/oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm

two。 Download the installation package for CDH5.16.1 at:

Http://archive.cloudera.com/cdh6/parcels/5.16.1/CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcelhttp://archive.cloudera.com/cdh6/parcels/5.16.1/CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel.sha1http://archive.cloudera.com/cdh6/parcels/5.16.1/manifest.json

3. Download the seven rpm packages needed for the Cloudera Manager installation locally, place them in the same directory, and execute the createrepo command to generate RPM metadata.

[root@ip-172-31-13-38 cm5.16.1] # lltotal 1019160 root root 9864584 Nov 27 14:40 cloudera-manager-agent-5.16.1-1.cm5161.p0.1.el7.x86room64.rpmAfter-1 root root 789872988 Nov 27 14:40 cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86q64.rpmAfter-1.cm5161.p0.1.el7.x86 cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86- -1 root root 8704 Nov 27 14:40 cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86y64.rpmWhen rwkuhashi-1 root root 10612 Nov 27 14:40 cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86" 64.rpmWhure RWMURT-1 root root 30604172 Nov 27 14:40 RPM-1. Cm5161.p0.1.el7.x86_64.rpm-rw-r--r-- 1 root root 71204325 Nov 27 14:40 jdk-6u31-linux-amd64.rpm-rw-r--r-- 1 root root 142039186 Nov 27 14:40 oracle-j2sdk1.7-1.7.0+update67-1.x8664.rpm [root @ ip-172-31-13-38 cm5.16.1] # createrepo .spawning worker 0 with 2 pkgsSpawning worker 1 with 2 pkgsSpawning worker 2 pkgsSpawning worker 3 with 1 pkgsWorkers FinishedSaving Primary metadataSaving file lists metadataSaving other metadataGenerating sqlite DBsSqlite DBs complete

4. Configure the Web server

Move the above cdh6.16.1/cm5.16.1 directory to the / var/www/html directory so that users can access these rpm packages through HTTP.

[root@ip-172-31-13-38] # mv cm5.16.1/ cdh6.16.1/ / var/www/html/

Verify that the browser can access properly

5. Make the repo source of Cloudera Manager

[root@ip-172-31-13-38] # vim / etc/yum.repos.d/ cm.repo [cmrepo] name = cm_repobaseurl = http://172.31.13.38/cm5.16.1enable = truegpgcheck = false [root @ ip-172-31-13-38 yum.repos.d] # yum repolistLoaded plugins: amazon-id, rhui-lb Search-disabled-reposrepo id repo name statuscmrepo cm_repo 7rhui-REGION-client-config-server-7/x86_64 Red Hat Update Infrastructure 2.0 Client C 1rhui- REGION-rhel-server-releases/7Server/x86_64 Red Hat Enterprise Linux Server 7 (RPMs) 20668rhui-REGION-rhel-server-rh-common/7Server/x86_64 Red Hat Enterprise Linux Server 7 RH Commo 233repolist: 20909

6. Verify installation of JDK

Yum-y install oracle-j2sdk1.7-1.7.0+update67-1

3.2. Install Cloudera Manager Server

1. Install Cloudera Manager Server through yum

Yum-y install cloudera-manager-server

two。 Initialize the database

[root@ip-172-31-13-38 yum.repos.d] # / usr/share/cmf/schema/scm_prepare_database.sh mysql cm cm passwordJAVA_HOME=/usr/java/jdk1.7.0_67-clouderaVerifying that we can write to / etc/cloudera-scm-serverCreating SCM configuration file in/ etc/cloudera-scm-serverExecuting: / usr/java/jdk1.7.0_67-cloudera/bin/java-cp / usr/share/java/mysql-connector-java.jar:/usr / share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor / etc/cloudera-scm-server/db.properties com.cloudera.cmf.db. [main] DbCommandExecutor INFO Successfully connected to database.All done Your SCM database is configured correctly!

3. Start Cloudera Manager Server

[root@ip-172-31-13-38] # systemctl start cloudera-scm-server

4. Check if the port is listening

[root@ip-172-31-13-38 ~] # netstat-lnpt | grep 7180tcp 0 0 0.0 0. 0. 0. 0. 0. 0 LISTEN 3331/java

5. Access CM through http://13.250.14.115:7180/cmf/login

4.CDH installation 4.1.CDH Cluster installation Wizard

1.admin/admin logs in to CM

two。 Agree to the license agreement and click continue

3. Select 60 for trial and click to continue

4. Click "continue"

5. Enter the ip or name of the host, click search to find the host, and click continue

6. Click "continue"

Note that the CDH versions listed here are the default at the beginning of the system, and come from the download source repository of the Cloudera public network. Here, we need to modify the installation source of CDH.

7. Select using parcel, click "more options", click "-" to delete all other addresses, enter

Http://172.31.13.38/cdh6.16.1, click "Save changes"

After saving the changes, when you go back to the previous page, you will see the source of the CDH5.16.1 of http that we prepared before. If it does not show up, there may be something wrong with the configuration of the http source. Please refer to the previous steps to check carefully.

8. Select a custom repository and enter the http address of cm

9. Click "continue" to proceed to the next step to install jdk

10. Click "continue" to proceed to the next step, which defaults to multi-user mode without any check box

11. Click "continue" to proceed to the next step to configure the ssh account password

twelve。 Click "continue" to proceed to the next step, install Cloudera Manager related to each node

13. Click "continue" to proceed to the next step to install cdh to each node

14. Click "continue" to proceed to the next host check to ensure that all check items are passed.

Click finish to enter the service installation wizard.

4.2. Cluster setup installation Wizard

1. Select the services to be installed

two。 Click "continue" to enter the cluster role assignment

3. Click "continue" to proceed to the next step, test the database connection

4. If the test is successful, click "continue" to enter the directory settings. Use the default default directory here and modify the directory according to the actual situation.

5. Click "continue" to start each service.

6. Installation succeeded

7. Go to the CM management interface home page after successful installation

5.Kudu installation

CDH packages Kudu directly from May 13, eliminating the need to download additional Parcel and csd files. CDH5.13 integrates with Kudu1.5,CDH5.16.1 and integrates with Kudu1.7.

1. Install Kudu1.7 through CM

Add Kudu service

two。 Select Master and Tablet Server

3. Configure the appropriate directories. Note: both Master and Tablet should have multiple data directories (fs_data_dir) depending on the actual situation, to improve concurrent read and write, thereby improving Kudu performance.

4. Start the Kudu service

5. Installation completed

6. Configure Impala

Go back to the home page and restart Impala and Hue

At this point, CDH5.16.1+Kudu1.7 is installed.

The above is all the contents of the article "how to install CDH5.16.1 in Redhat7.4". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.