Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to build the big data platform based on Ambari. The content of the article is of high quality, so the editor will share it with you for reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

Similar to CDH deployment, the steps are divided into ambari deployment and hdp deployment. Take one as an example (memory > 6 GB, disk partition / at least 40 G, 10 GB), and subsequent nodes can join the cluster by expanding capacity.

1. Prepare the installation package

Need to download jdk-8u144-linux-x64.tar.gz,ambari-xx-centos7.tar.gz,HDP-xx-centos7-rpm.tar.gz,HDP-UTILS-xx-centos7.tar.gz locally (the rpm package is very large), link:

Http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.1.5/ambari-2.6.1.5-centos7.tar.gzhttps://download.csdn.net/download/ljk168/10351315http://public-repo-1.hortonworks.com/HDP-GPL/centos7/2.x/updates/2.6.4.0/HDP-GPL-2.6.4.0-centos7-rpm.tar.gzhttp://public-repo- 1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.4.0/HDP-2.6.4.0-centos7-rpm.tar.gz2 、 Mainframe ready.

Modify the network settings of the virtual machine, disable SElinux, modify the host name, turn off the firewall, ssh secret-free, turn on http service, and set clock synchronization; the requirement for the network is: in NAT mode, all hosts are in the same network segment and can access the external network. Modify IP, gateway, DNS

Vim / etc/sysconfig/network-scripts/ifcfg-ens33

Modify the dhcp auto-allocation of ifcfg-en33 to static addressing and add ip/ mask / gateway configuration; (no matter how the gateway is configured, do not configure 192.168.x.1, because IP 1 is the IP of VMnet8 network card)

TYPE= "Ethernet" PROXY_METHOD= "none" BROWSER_ONLY= "no" BOOTPROTO= "static" DEFROUTE= "IPV4_FAILURE_FATAL=" no "IPV6INIT=" yes "IPV6_AUTOCONF=" yes "IPV6_DEFROUTE=" yes "IPV6_FAILURE_FATAL=" no "IPV6_ADDR_GEN_MODE=" stable-privacy "NAME=" ens33 "UUID=" 5ed941c2-deb4-48ae-a76d-838503c8683c "DEVICE=" ens33 "ONBOOT=" yes "IPADDR=" 192.168.242.111 "GATWAY=" 192.168.242.2 "NETMASK=" 255.255.255.0 "DNS1=" 192.168.242.2 "NETMASK=" 255.255.255.0 "DNS1=" 192.168.242.111 "GATWAY=" 192.168.242.2 "NETMASK=" 255.255.255.0 "DNS1=" 192.168.242.2 "

After the configuration is completed, restart the Nic settings via service network restart, but still cannot connect to the public network (ping www.baidu.com). You can add the default dns.

Route add default gw 192.168.242.2

Secondly, add host and host name mapping (/ etc/hosts), turn off firewall, turn off SElinux, set multi-host privacy, and enable http service.

192.168.242.134 hadoop1

# disable the firewall, the previous version of centos7 systemctl disable firewalldsystemctl stopfirewalld# closes the firewall, and the restart of version systemctl disable firewalld.service# takes effect after centos7

# vi / etc/selinux/configSELINUX=disabledSELINUXTYPE=targeted

How to keep the host secret free:

Take host A/B/C as an example, set secret-free login method: ① executes ssh-keygen-t rsa on each machine. Hitting enter three times will generate a hidden .ssh directory under the home directory / home/csap. First, you need to assign permissions to this directory: chmod 700.ssh. There are two files in this directory: id_rsa private key and id_rsa.pub public key. ② creates the authorized_keys file in this directory and empowers: cd .sshtouch authorized_keyschmod 644 authorized_keys ③ take machine An as an example, if you do not know the password of other machines, such as binding 4A, paste your own id_rsa.pub public key and the public keys of the other two machines into the authorized_keys file, and then copy the file to the other two machines. If all machines have the public keys of other machines, you can complete the ssh secret-free login; if ④ knows the passwords of other machines, you can directly execute this and write the public key to the authorized_keys file of other machines: ssh-copy-id-I ~ /. Ssh/id_rsa.pub?root@127.0.0.1

Clock synchronization (ensure that other nodes have the same clock as the ambari master node):

1. All machines install ntp: yum-y install ntp2, master node configuration clock synchronizes with itself: vim / etc/ntp.conf, delete other server, add: server 127.127.1.0 # local clockfudge 127.127.1.0 stratum 103, other machines synchronize the time of master node, vim / etc/ntp.conf Join: server xxx.xxx.xxx.xx4, restart the ntp service systemctl restart ntpd or service ntpd restartsystemctl status ntpd or service ntpd status5 of all machines, verify that all nodes execute ntpq-p, and the * sign on the left indicates that the synchronization is successful. 6. If unsuccessful, / usr/sbin/ntpdate stdtime.gov.hkntpdate xxx.xxx.xxx.xxx manually synchronizes the time

Enable the http service, and the cm and cdh directories will be created automatically under the / var/www/html/ directory.

Yum-y install httpdsystemctl start httpd or service httpd start

Install jdk and configure JAVA_HOME

# Uninstall other jdk and use the following command to view the rpm-qa of all current Java environments | grep java# uses the following command to delete all Java installation packages rpm-e-- nodeps package name # install jdkmkdir / data/jdkcd / data/jdktar-xvf jdk-8u144-linux-x64.tar.gz# to modify the environment variables Configure the path of java vi ~ / .bash_profile# add configuration: JAVA_HOME=/data/jdk/jdk1.8.0_144PATH=$PATH:$HOME/bin:$ {JAVA_HOME} / bin:$ {JAVA_HOME} / sbin# to make effective source ~ / .bash_profile#java-version instruction check whether it works 3, configure mariaDB. As Metabase for hive and ambari # install database yum-y install mariadb-server

# start systemctl start mariadb.servicesystemctl enable mariadb.service

# the first step is to set the password. You will be prompted to enter the password, enter directly, set the password 123456, and enter mysql_secure_installation all the way.

# configuration database mysql-u root-p 123456

# (1) create ambari database and its users: create database ambari;CREATE USER 'ambari'@'%'IDENTIFIED BY' 123456 clients *. * TO 'ambari'@'%';FLUSH PRIVILEGES

# (2) create the Hive database and the user executes the following statement: create database hive;CREATE USER 'hive'@'%'IDENTIFIED BY' 123456 alternate Grant ALL PRIVILEGES ON. TO 'hive'@'%';FLUSH PRIVILEGES

# (3) create the Oozie database and the user executes the following statement: create database oozie;CREATE USER 'oozie'@'%'IDENTIFIED BY' 123456 alternate Grant ALL PRIVILEGES ON. TO 'oozie'@'%';FLUSH PRIVILEGES

# finally, install the jdbcyum-y mysql-connector-java# jdbc for the database: / usr/share/java/mysql-connector-java.jar4, upload the installation package, create a local yum source, and start installing ambari.

Configure the yum source (all nodes), delete all repo files under / etc/yum.repos.d/, clear the previous cache yum clean all, and check "connected" in the options set by the virtual machine:

# Mount the file (note that the file used here is effective at that time and will expire after shutdown. It is recommended to set permanent mount files): mount / dev/cdrom / mnt/

# set your own yum source vi / etc/yum.repos.d/my.repo

[myrepo] baseurl= file:///mntenabled=1gpgcheck=0

The RPM files downloaded to the local hdp and ambari are uploaded to the / var/wwww/html/ambari directory. Some files are too large (> 4G). The rz command of xshell cannot upload directly. Please refer to the post: https://blog.csdn.net/FLawiet/article/details/89597389

# extract the installation package cd / var/www/html/ambari/tar-zxvf ambari-2.6.1.5-centos7.tar.gz-C / var/www/html/ambaritar-zxvf HDP-2.6.4.0-centos7-rpm.tar.gz-C / var/www/html/ambarimkdir / var/www/html/ambari/HDP-UTILS/tar-zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz / var/www/html/ambari/HDP-UTILS/

# after decompression, you need to delete all the html files in the directory, and then resume the card owner to avoid errors; cd / var/www/html/ambari/find. -name * .html# html files are concentrated in one folder; cd. / HDP/centos7/2.6.4.0-91/rm * .html

# after decompression, ambari.repo, hdp.repo and hdp-utils.repo files are generated in the lowest level of the folder # after each url is modified, it is copied to the / etc/yum.repos.d/ directory as the local yum source

# ① modify ambari.repo in / etc/yum.repos.d folder to ensure that baseurl can be accessed by http Cat ambari.repo # is as follows # VERSION_NUMBER=2.6.1.5-3 [ambari-2.6.1.5] name=ambari Version-ambari-2.6.1.5baseurl= http://192.168.242.111/ambari/ambari/centos7/2.6.1.5-3/gpgcheck=1gpgkey=http://192.168.242.111/ambari/ambari/centos7/2.6.1.5-3/RPM-GPG-KEY/RPM-GPG-KEY-Jenkinsenabled=1priority=1

# ② modify hdp.repo in / etc/yum.repos.d folder to ensure that baseurl can be accessed by http Cat hdp.repo # is as follows # VERSION_NUMBER=2.6.4.0-91 [HDP-2.6.4.0] name=HDP Version-HDP-2.6.4.0baseurl= http://192.168.242.111/ambari/HDP/centos7/2.6.4.0-91/gpgcheck=1gpgkey=http://192.168.242.111/ambari/HDP/centos7/2.6.4.0-91/RPM-GPG-KEY/RPM-GPG-KEY-Jenkinsenabled=1priority=1

[HDP-UTILS-1.1.0.22] name=HDP-UTILS Version-HDP-UTILS-1.1.0.22baseurl= http://192.168.242.111/ambari/HDP-UTILS/centos7/1.1.0.22# baseurl= http://192.168.242.111/ambari/HDP-UTILSgpgcheck=1gpgkey=http://192.168.242.111/ambari/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins# gpgkey= http://192. 168.242.111/ambari/HDP-UTILS/RPM-GPG-KEY/RPM-GPG-KEY-Jenkinsenabled=1priority=1

Start installing ambari-server.

# install yum install-y ambari-server# and configure ambari-server setup

The configuration prompts for the login passwords for the JAVA_HOME and Ambari Metabase:

Log in to the ambari database and initialize the database

Mysql-uroot-p123456use ambarisource / var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sqlshow tables

After initializing the database, start ambari-server start. If you report an error in the middle, you can read the log: / var/log/ambari-server/ambari-server.log. Generally, it is a database permission problem, similar to cm of CDH, and web access (admin,admin) after successful installation.

5. Start using ambari interface operation and configure HDP.

Follow the instructions and select the location of the rmp package for HDP:

Possible error: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password), see if the authorized_keys file has the server's own public key content. Error: NetUtil.py:124-Server at https://hadoop1:8440 is not reachable, sleeping for 10 seconds... ", None) reference: https://blog.csdn.net/qq_41805514/article/details/88871477

Component installation, due to limited resources, only hdfs,mr,yarn,zk and spark components are installed here as an example.

On the Ambari-based big data platform is how to share here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.