Shulou(Shulou.com)06/01 Report--

This article is to share with you about how Amabari builds Hadoop clusters. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

I. system environment

1. System version

[root@manager ~] # cat / etc/centos-releaseCentOS Linux release 7.4.1708 (Core)

two。 Host planning

Hostname IP address role manager192.168.10.131

Ambari-servervm1

192.168.10.128agent

Vm2192.168.10.129agentvm3192.168.10.130agent

3. Software version

Ambari-2.4.3.0-centos7.tar.gzHDP-2.5.3.0-centos7-rpm.tar.gzHDP-UTILS-1.1.0.21-centos7.tar.gz

Second, environment preparation (to be configured on all nodes)

1. Modify hostname

two。 Firewall and selinux

[root@manager] # systemctl status firewalld ● firewalld.service-firewalld-dynamic firewall daemon Loaded: loaded (/ usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled) Active: inactive (dead) Docs: man:firewalld (1) [root@manager] # getenforce Disabled [root@manager] #

3. Configure the hosts file

[root@manager ~] # cat / etc/hosts127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain6192.168.10.128 vm1192.168.10.129 vm2192.168.10.130 vm3192.168.10.131 manager

4. Configure SSH password-free login

5. Turn off the THP service for linux

# add transparent_hugepage=never after the line GRUB_CMDLINE_LINUX The following is [root@manager ~] # cat / etc/default/grub GRUB_TIMEOUT=5GRUB_DISTRIBUTOR= "$(sed's, release. * $) G'/ etc/system-release) "GRUB_DEFAULT=savedGRUB_DISABLE_SUBMENU=trueGRUB_TERMINAL_OUTPUT=" console "GRUB_CMDLINE_LINUX=" crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet transparent_hugepage=never "GRUB_DISABLE_RECOVERY=" true "# execute the following command [root@manager ~] # grub2-mkconfig-o / boot/grub2/grub.cfg [root@manager ~] # systemctl disable tuned# and display the following [root@manager ~] # cat / sys/kernel/mm/ after restart Transparent_hugepage/enabledalways madvise [never] [root@manager ~] # grep AnonHugePages / proc/meminfo AnonHugePages: 0 kB [root@manager ~] #

6. Configure UMASK

[root@manager ~] # umask0022 [root@manager ~] # umask0022

7. Configure time synchronization NTP

[root@manager ~] # yum-y install ntp [root@manager ~] # systemctl start ntpd [root@manager ~] # systemctl enable ntpd

8. Install JDK and configure the JAVA_HOME environment variable

3. Install Ambari cluster (only configured on manager)

1. Install the HTTP service

[root@manager ~] # yum install-y httpd [root@manager ~] # systemctl start httpd [root@manager ~] # systemctl enable httpd

two。 Install MySQL

3. Configure MySQL

# mysql has a password complexity requirement If you do not need the password policy to add the following configuration in the / etc/my.cnf file and restart mysql, you can disable validate_password = offsystemctl restart mysqld# to view the initial default password of mysql [root@manager ~] # cat / var/log/mysqld.log | grep password2017-12-13T01:49:55.629403Z 1 [Note] A temporary password is generated for root@localhost: 3aYp&Zsqq7Ov# uses the initial password to enter mysql to modify the password ALTER USER 'root'@'localhost' IDENTIFIED BY' 123456' # create ambari database and user CREATE DATABASE ambari CHARACTER SET utf8 in mysql; CREATE USER 'ambari'@'%'IDENTIFIED BY' 123456 create create USER 'ambari'@'local'IDENTIFIED BY' 123456 create Grant ALL PRIVILEGES ON *. * TO 'ambari'@'%';FLUSH PRIVILEGES;# to create hive and other databases are similar

4. Download Ambari and HDP

Wget-nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.4.3.0/ambari-2.4.3.0-centos7.tar.gzwget-nv http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/HDP-2.5.3.0-centos7-rpm.tar.gzwget-nv http://public-repo-1.hortonworks.com/HDP-UTILS -1.1.0.21/repos/centos7/HDP-UTILS-1.1.0.21-centos7.tar.gz

5. Unzip it to httpd directory

[root@manager ~] # mkdir / var/www/html/ambari [root@manager ~] # mkdir / var/www/html/ambari/HDP-UTILStar zxvf ambari-2.4.3.0-centos7.tar.gz-C / var/www/html/ambaritar zxvf HDP-2.5.3.0-centos7-rpm.tar.gz-C / var/www/html/ambaritar zxvf HDP-UTILS-1.1.0.21-centos7.tar.gz-C / var/www/html/ambari/HDP-UTILS

6. Install tools related to local source production

[root@manager ~] # yum install-y yum-utils createrepo yum-plugin-priorities [root@manager ~] # vim / etc/yum/pluginconf.d/priorities.conf [main] enabled = 1gpgcheck=0 [root@manager data] # cd / var/www/html/ambari [root@manager ambari] # createrepo. /

7. Configure local source

[root@manager ~] # cat / etc/yum.repos.d/ambari.repo [ambari-2.4.3.0] name=ambari-2.4.3.0baseurl= http://manager/ambari/ambari/centos7/gpgcheck=1gpgkey=http://manager/ambari/ambari/centos7/RPM-GPG-KEY/RPM-GPG-KEY-Jenkinsenabled=1priority=1[root@manager ~] # cat / etc/yum.repos.d/hdp.repo [HDP-2.5.3.0] name=HDPbaseurl= http://manager/ambari/HDP/centos7/gpgcheck=1gpgkey=http : / / manager/ambari/HDP/centos7/RPM-GPG-KEY/RPM-GPG-KEY-Jenkinsenabled=1priority=1 [HDP-UTILS-1.1.0.21] name=HDP-UTILSbaseurl= http://manager/ambari/HDP-UTILS/gpgcheck=1gpgkey=http://manager/ambari/HDP-UTILS/RPM-GPG-KEY/RPM-GPG-KEY-Jenkinsenabled=1priority=1[root@manager ~] # yum clean all [root@manager ~] # yum makecache# View the repository of ambari and hdp [root@manager ~] # yum repolistrepo id Repo name statusHDP-2.5.3.0 HDP 200HDP-UTILS-1 .1.0.21 HDP-UTILS 64ambari-2.4.3.0 ambari-2.4.3.0

8. Install ambari and mysql drivers

[root@manager ~] # yum install-y ambari-server [root@manager ~] # yum install-y mysql-connector-java

9. Configure ambari

[root@manager] # ambari-server setupUsing python / usr/bin/pythonSetup ambari-serverChecking SELinux...SELinux status is' disabled'Customize user account for ambari-server daemon [yzone] (n)? YEnter user account for ambari-server daemon (root): rootAdjusting ambari-server permissions and ownership...Checking firewall status...Checking JDK...Do you want to change Oracle JDK [yPao] (n)? Y [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8 [2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7 [3] Custom JDK====Enter choice (1): 3WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.Path to JAVA_HOME: / usr/local/jdkValidating JDK on Ambari Server...done.Completing setup...Configuring database...Enter advanced database configuration [yzone] (n)? YConfiguring database...====Choose one of the following options: [1]-PostgreSQL (Embedded) [2]-Oracle [3]-MySQL / MariaDB [4]-PostgreSQL [5]-Microsoft SQL Server (Tech Preview) [6]-SQL Anywhere [7]-BDB====Enter choice (3): 3Hostname (localhost): Port (3306): Database name (ambari): Username (ambari): Enter Database Password (123456): Configuring ambari database...Copying JDBC drivers to server resources...Configuring remote database connection properties...WARNING: Before starting Ambari Server You must run the following DDL against the database to create the schema: / var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sqlProceed with configuring remote database connection properties [yzone] (y)? Y Extracting system views.Adjusting ambari-server permissions and ownership...Ambari Server 'setup' completed successfully.

10. Import data in the ambari database using the msyql user ambari

[root@manager ~] # mysql-u ambari- p123456mysql > use ambariDatabase changedmysql > source / var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

11. Start ambari

[root@manager ~] # ambari-server start# enter the ambari address http://192.168.10.131:8080 in the browser. Thank you for reading! This is the end of the article on "how to build a Hadoop cluster in Amabari". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.