Shulou(Shulou.com)06/01 Report--

This article is to share with you the content of the sample analysis of the Hadoop high-availability build. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

Hadoop high availability build in super detail

Experimental environment

1. Install jdk

two。 Modify hostname

3. Modify hosts mapping and configure ssh secret-free login

4. Set up time synchronization

5. Install hadoop to / opt/data directory

6. Modify hadoop configuration file

7.zookeeper cluster installation configuration

8. Start the cluster

Installation steps

Experimental environment master:192.168.10.131slave1:192.168.10.129slave2:192.168.10.130 operating system ubuntu-16.04.3hadoop-2.7.1zookeeper-3.4.8 installation step 1. Install jd

Install jdk to the opt directory

Tar-zvxf jdk-8u221-linux-x64.tar.gz

Configure environment variables

Vim etc/profile#jdkexport JAVA_HOME=/opt/jdk1.8.0_221export JRE_HOME=$ {JAVA_HOME} / jreexport CLASSPATH=.:$ {JAVA_HOME} / lib:$ {JRE_HOME} / libexport PATH=$ {JAVA_HOME} / bin:$PATHsource etc/profile2. Modify hostname

Modify the three virtual machines to master, slave1 and slave2 respectively

Vim / etc/hostname3. Modify hosts mapping and configure ssh secret-free login

To modify the hosts file, each host needs to do the following

Vim / etc/hosts192.168.10.131 master192.168.10.129 slave1192.168.10.130 slave2

Configure ssh non-secret

First, you need to turn off the firewall.

1. Check the port open status sudo ufw status2, open a port, for example, I open 8381sudo ufw allow 83813, enable firewall sudo ufw enable4, turn off firewall sudo ufw disable 5, restart firewall sudo ufw reload 6, disable an external port such as 80 sudo ufw delete allow 807, and check port ip netstat-ltn.

During the startup process of the cluster, ssh is required to log in remotely to another host. In order to avoid entering the password of the other host each time, we need to configure password-free login (press enter for prompt operations)

Ssh-keygen-t rsa

Copy the public key of each host to yourself and other hosts

Ssh-copy-id-I ~ / .ssh/id_rsa.pub root@masterssh-copy-id-I ~ / .ssh/id_rsa.pub root@slave1ssh-copy-id-I ~ / .ssh/id_rsa.pub root@slave24. Set up time synchronization

Install the ntpdate service

Apt-get install ntpdate

Modify ntp configuration file

Vim / etc/ntp.conf# / etc/ntp.conf, configuration for ntpd; see ntp.conf (5) for helpdriftfile / var/lib/ntp/ntp.drift# Enable this if you want statistics to be logged.#statsdir / var/log/ntpstats/statistics loopstats peerstats clockstatsfilegen loopstats file loopstats type day enablefilegen peerstats file peerstats type day enablefilegen clockstatsfile clockstats type day enable# Specify one or more NTP servers.# Use servers from the NTP Pool Project. Approved by Ubuntu Technical Board# on 2011-02-08 (LP: # 104525) See http://www.pool.ntp.org/join.html for# more information.#pool 0.ubuntu.pool.ntp.org iburst#pool 1.ubuntu.pool.ntp.org iburst#pool 2.ubuntu.pool.ntp.org iburst#pool 3.ubuntu.pool.ntp.org iburst# Use Ubuntu's ntp server as a fallback.#pool ntp.ubuntu.com# Access control configuration; see / usr/share/doc/ntp-doc/html/accopt.html for# details. The web page # might also be helpful.## Note that "restrict" applies to both servers and clients, so a configuration# that might be intended to block requests from certain clients could also end# up blocking replies from your own upstream servers.# By default, exchange time with everybody, but don't allow configuration.restrict-4 default kod notrap nomodify nopeer noquery limitedrestrict-6 default kod notrap nomodify nopeer noquery limited# Local users may interrogate the ntp server more closely.restrict 127.0.0.1restrict:: Mr. Needed for adding pool entriesrestrict source notrap nomodify noquery# Clients from this (example!) Subnet have unlimited access, but only if# cryptographically authenticated.# allows local area network devices to synchronize time with this server. But refused to let them change the time on the server # restrict 192.168.10.131 mask 255.255.255.0 nomodify notrust#statsdir / var/log/ntpstats/statistics loopstats peerstats clockstatsfilegen loopstats file loopstats type day enablefilegen peerstats file peerstats type day enablefilegen clockstatsfile clockstats type day enable# Specify one or more NTP servers.# Use servers from the NTP Pool Project. Approved by Ubuntu Technical Board# on 2011-02-08 (LP: # 104525) See http://www.pool.ntp.org/join.html for# more information.#pool 0.ubuntu.pool.ntp.org iburst#pool 1.ubuntu.pool.ntp.org iburst#pool 2.ubuntu.pool.ntp.org iburst#pool 3.ubuntu.pool.ntp.org iburst# Use Ubuntu's ntp server as a fallback.#pool ntp.ubuntu.com# Access control configuration; see / usr/share/doc/ntp-doc/html/accopt.html for# details. The web page # might also be helpful.## Note that "restrict" applies to both servers and clients, so a configuration# that might be intended to block requests from certain clients could also end# up blocking replies from your own upstream servers.# By default, exchange time with everybody, but don't allow configuration.restrict-4 default kod notrap nomodify nopeer noquery limitedrestrict-6 default kod notrap nomodify nopeer noquery limited# Local users may interrogate the ntp server more closely.restrict 127.0.0.1restrict:: Mr. Needed for adding pool entriesrestrict source notrap nomodify noquery# Clients from this (example!) Subnet have unlimited access, but only if# cryptographically authenticated.# allows local area network devices to synchronize time with this server. But refused to let them change the time on the server # restrict 192.168.10.131 mask 255.255.255.0 nomodify notrustrestrict 192.168.10.129 mask 255.255.255.0 nomodify notrustrestrict 192.168.10.130 mask 255.255.255.0 nomodify notrust# allows the upper time server to modify the local time # restrict times.aliyun.com nomodify#restrict ntp.aliyun.com nomodify#restrict cn.pool.ntp.org nomodify# defines the time to be synchronized Time server server 192.168.10.131 perfer#server times.aliyun.com iburst prefer # prefer is expressed as priority Indicates that the server time # server ntp.aliyun.com iburst#server cn.pool.ntp.org iburst#logfile / var/log/ntpstats/ntpd.log # defines the ntp log directory # pidfile / var/run/ntp.pid # defines the pid path # If you want to provide time to your local subnet, change the next line.# (Again, the address is an example only.) # broadcast 192.168.123.25 synchronization If you want to listen to time broadcasts on your local subnet De-comment the# next lines. Please do this only if you trust everybody on the networking disable auth#broadcastclient#Changes recquired to use pps synchonisation as explained in documentation:# http://www.ntp.org/ntpfaq/NTP-s-config-adv.htm#AEN3918#server 127.127.8.1 mode 135prefer # Meinberg GPS167 with PPS#fudge 127.127.8.1 time1 0.0042 # relative to PPS for my hardware#server 127.127.22.1 # ATOM (PPS) # Fudge 127.127.22.1 flag3 1 # enable PPS APIserver 127.127.1.0fudge 127.127.1.0 stratum 10

Start the ntpd service and view the ntp synchronization status

Service ntpd start # start ntp service ntpq-p # observe time synchronization status ntpstat # View time synchronization results

Restart the service to synchronize with the master host time

/ etc/init.d/ntp restartntpdate 192.168.10.1315. Install hadoop to / opt/data directory

Create a new Data directory under the / opt directory

Cd / optmkdir Data

Download and extract hadoop to the / opt/data directory

Wget https://archive.apache.org/dist/hadoop/common/hadoop-2.7.1/tar-zvxf hadoop-2.7.1.tar / opt/data

Configure environment variables

# HADOOPexport HADOOP_HOME=/opt/Data/hadoop-2.7.1export HADOOP_CONF_DIR=$ {HADOOP_HOME} / etc/hadoopexport PATH=$ {HADOOP_HOME} / bin:$ {HADOOP_HOME} / sbin:$PATHexport HADOOP_YARN_HOME=$HADOOP_HOME6. Modify hadoop configuration file

File directory hadoop-2.7.1/etc/hadoop

Modify hadoop-env.sh

Export JAVA_HOME=/opt/jdk1.8.0_221

Modify core-site.xml

Fs.defaultFS hdfs://ns1/ hadoop.tmp.dir / opt/Data/hadoop-2.7.1/tmp ha.zookeeper.quorum slave1:2181,slave2:2181 ipc.client.connect.max.retries 100 Indicates the number of retries a client will make to establish a server connection.

Modify hdfs-site.xml

Dfs.nameservices ns1 dfs.ha.namenodes.ns1 nn1,nn2 dfs.namenode.rpc-address.ns1.nn1 master:9820 dfs.namenode.http-address.ns1.nn1 master:9870 dfs.namenode.rpc-address.ns1.nn2 slave1:9820 dfs.namenode.http-address.ns1.nn2 slave1:9870 dfs.namenode.shared.edits.dirqjournal://master:8485;slave1:8485 Slave2:8485/ns1 dfs.journalnode.edits.dir / opt/Data/hadoop-2.7.1/journal dfs.ha.automatic-failover.enabled true dfs.client.failover.proxy.provider.ns1org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider dfs.ha.fencing.methods sshfence shell (/ bin/true) dfs.ha. Fencing.ssh.private-key-files / root/.ssh/id_rsa dfs.ha.fencing.ssh.connect-timeout 30000 dfs.namenode.name.dir / opt/Data/hadoop-2.7.1/data/name dfs.datanode.data.dir / opt/Data/hadoop-2.7.1/data/data dfs.replication 2 dfs.webhdfs.enabled true

Modify mapred-site.xml

Change the file name to mapred-site.xmlcp mapred-queues.xml.template mapred-site.xml mapreduce.framework.name yarn

Modify yarn-site.xml

Yarn.nodemanager.aux-services mapreduce_shuffle yarn.resourcemanager.ha.enabled true yarn.resourcemanager.cluster-id cluster1 yarn.resourcemanager.ha.rm-ids rm1,rm2 yarn.resourcemanager.hostname.rm1 master yarn.resourcemanager.hostname.rm2 slave1 yarn.resourcemanager.zk-address slave1:2181 Slave2:2181 yarn.nodemanager.vmem-check-enabled false Whether virtual memory limits will be enforced for containers yarn.nodemanager.vmem-pmem-ratio 8 Ratio between virtual memory to physical memory when setting memory limits for containers

Modify slave

Masterslave1slave27.zookeeper cluster installation configuration

Download and extract zookeeper-3.4.8.tar.gz

Wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.8/zookeeper-3.4.8.tar.gztar-zvxf zookeeper-3.4.8.tar.gz / opt/Data

Modify the configuration file

# zookeeperexport ZOOKEEPER_HOME=/opt/Data/zookeeper-3.4.8export PATH=$PATH:$JAVA_HOME/bin:$ZOOKEEPER_HOME/bin

Enter the conf directory and copy zoo-sample.cfg to zoo.cfg

Cp zoo-sample.cfg zoo.cfg

Modify zoo.cfg

DataDir=/opt/Data/zookeeper-3.4.8/tmp / / you need to create a new tmp directory server.1=master:2888:3888server.2=slave1:2888:3888server.3=slave2:2888:3888 under the zookeeper-3.4.8 directory

Create a myid file in the tmp directory

Vim myid 1 / / other hosts need to modify the number 2Jing 38. Start the cluster

Format the master host namenode. Enter this command in the / etc/hadoop directory

Hadoop namenode-format

Copy the Data directory to the other two hosts

Scp-r / opt/Data root@slave1:/optscp-r / opt/Data root@slave2:/opt

Start zookeeper, and all nodes execute

Hadoop-daemon.sh start zkfc

Format zookeeper, all nodes execute

Hdfs zkfc-formatZK

Starting the journalnode,namenode backup node is the same (executed under the hadoop-2.7.1 directory)

Hadoop-daemon.sh start journalnode

Start the cluster

Start-all.sh

View Port

Netstat-ntlup # can view the ports occupied by the server

View process jps

Thank you for reading! This is the end of this article on "sample Analysis of Hadoop High availability Building". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.