Shulou(Shulou.com)05/31 Report--

This article mainly introduces how to configure hadoop under the HA mechanism, which is very detailed and has a certain reference value. Interested friends must read it!

1. The HA mechanism of hdfs

One NameNode server is ACTIVE and one is STANDBY. Using zkfc through Qjournal (log management system) (based on zookeeper failover control)

How to prevent brain fissure: when ACTIVE dies, the zkfc of STANDBY will instruct ssh kill-9 namenode to return 0 (successful kill, 1 kill failure) and then notify the second namenode to switch status. If there is no return code (timeout does not respond), start proweroff.sh (user-defined script) to power off the first namenode, return 0 to execute normally, and then switch.

When the first machine is powered off (meaning that both the active namenode and zkfc are powered off), the zkfc of the second will execute proweroff.sh.

Key words: 1. There are two namenode, active and standby.

2. There are two zkfc to monitor and manage the status of the two namenode

3. Metadata log edits is managed by a special log system-- qjoournal.

4. The functions of zkfc and qjournal depend on zookeeper services.

5. Zkfc has two mechanisms to prevent brain fissure during state switching-ssh and shell scripts.

II. Configuration of Hadoop cluster

Open the installation directory of hadoop. If you can't download it or use it, please see the link I provided at the beginning of the article. Please learn the basic configuration first. I mainly use 5 virtual machines here to simulate and build the whole cluster environment.

Main functions of hostname IP address

Ubuntu1 192.168.44.128 namenode zkfc ResourceManager

Ubuntu2 192.168.44.131 namenode zkfc ResourceManager

Ubuntu3 192.168.44.132 DataNode 、 NodeManager 、 JournalNode 、 QuorumPeerMain

Ubuntu4 192.168.44.133 DataNode 、 NodeManager 、 JournalNode 、 QuorumPeerMain

Ubuntu5 192.168.44.134 DataNode 、 NodeManager 、 JournalNode 、 QuorumPeerMain

We can configure ubuntu1 first, and then scp the rest to other machines, core-site.xml.

Fs.defaultFS hdfs://ns1/ hadoop.tmp.dir / home/admin1/hadoop/HAhadoop/tmp/hadoop ha.zookeeper.quorum ubuntu3:2181,ubuntu4:2181,ubuntu5:2181

Hdfs-site.xml

1 2 3 4 dfs.nameservices 5 ns1 6 7 8 9 dfs.ha.namenodes.ns110 nn1 Nn211 12 13 14 dfs.namenode.rpc-address.ns1.nn115 ubuntu1:900016 17 18 19 dfs.namenode.http-address.ns1.nn120 Ubuntu1:5007021 22 23 24 dfs.namenode.rpc-address.ns1.nn225 ubuntu2:900026 27 28 29 dfs.namenode.http-address.ns1.nn230 Ubuntu2:5007031 32 33 34 dfs.namenode.shared.edits.dir35 qjournal://ubuntu3:8485 Ubuntu4:8485 Ubuntu5:8485/ns136 37 38 39 dfs.journalnode.edits.dir40 / home/admin1/hadoop/HAhadoop/journaldata41 42 43 44 dfs.ha.automatic-failover.enabled45 True46 47 48 49 dfs.client.failover.proxy.provider.ns150 org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider51 52 53 54 dfs.ha. Fencing.methods55 56 sshfence 57 shell (/ bin/true) 58 59 60 61 62 dfs.ha.fencing.ssh.private-key-files63 / home/admin1/.ssh/id_rsa64 65 66 67 dfs.ha.fencing.ssh.connect-timeout68 3000069 70

Mapred-site.xml

1 2 3 4 mapreduce.framework.name5 yarn6 7

Yarn-site.xml

12 3 4 5 yarn.resourcemanager.ha.enabled 6 true 7 8 9 10 yarn.resourcemanager.cluster-id11 yrc12 13 14 15 yarn.resourcemanager.ha.rm-ids16 rm1 Rm217 18 19 20 yarn.resourcemanager.hostname.rm121 ubuntu122 23 24 yarn.resourcemanager.hostname.rm225 ubuntu226 27 28 29 yarn.resourcemanager.zk-address30 ubuntu3:2181 Ubuntu4:2181,ubuntu5:218131 32 33 yarn.nodemanager.aux-services34 mapreduce_shuffle35 36

Slaves

1 ubuntu3 2 ubuntu4 3 ubuntu5

And change the host name. Contents of sudo vi / etc/hostname ubuntu1 and hosts:

192.168.44.128 ubuntu1 192.168.44.131 ubuntu2 192.168.44.132 ubuntu3 192.168.44.133 ubuntu4 192.168.44.134 ubuntu5

Note: be sure to delete the original two days 127.0.1.1 this, a hosts file, there can not be the same hostname and different ip address, I started because of this, because of a long time, datanode just can not build up.

Startup process:

1. Start the zookeeper cluster and enter "enable Service" in ubuntu3,4,5: bin/zkServer.sh start to view status: bin/zkServer.sh status.

2. Start journalnode and enter sbin/hadoop-daemon.sh start journalnode in ubuntu3,4,5 to run jps command to verify that there are more JournalNode processes.

3. Format HDFS

Execute the command on ubuntu1: the bin/hdfs namenode-format# format will generate a file based on the hadoop.tmp.dir configuration in core-site.xml. Here I configure / HAhadoop/tmp, and then copy / HAhadoop/tmp to / HAhadoop/ in ubuntu2. Scp-r tmp/ ubuntu2:/home/admin1/hadoop/HAhadoop/

4. Format ZKFC (execute on ubuntu1) bin/hdfs zkfc-formatZK

5. Start HDFS (execute on ubuntu1) sbin/start-dfs.sh

6. Start YARN (execute on ubuntu1) sbin/start-yarn.sh

If it is started in ubuntu1, the default browsing window will be like this: ubuntu2 is Active and ubuntu1 is a Standby state

At this time, if you go to ubuntu2 and kill the namenode process, ubuntu1 will automatically switch to Active,ubuntu2 and become unreachable.

On the remaining three datanode, it would look like this:

The above is all the contents of the article "how to configure hadoop under the HA mechanism". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.