Shulou(Shulou.com)05/31 Report--

Editor to share with you the HA mechanism in HDFS and Hadoop cluster building example analysis, I believe that most people do not know much about it, so share this article for your reference, I hope you will learn a lot after reading this article, let's go to understand it!

A brief introduction to 1.HA

Background

Prior to Hadoop 2.0.0, the NameNode was a single point of failure (SPOF) in an HDFS cluster. Each cluster had a single NameNode, and if that machine or process became unavailable, the cluster as a whole would be unavailable until the NameNode was either restarted or brought up on a separate machine.

This impacted the total availability of the HDFS cluster in two major ways:

In the case of an unplanned event such as a machine crash, the cluster would be unavailable until an operator restarted the NameNode.

Planned maintenance events such as software or hardware upgrades on the NameNode machine would result in windows of cluster downtime.

The HDFS High Availability feature addresses the above problems by providing the option of running two redundant NameNodes in the same cluster in an Active/Passive configuration with a hot standby. This allows a fast failover to a new NameNode in the case that a machine crashes, or a graceful administrator-initiated failover for the purpose of planned maintenance.

This paragraph is written in Youdao Translation for a while. That is to say, how can the previous mechanism not work, and what should be improved? It is mainly to solve the single point of failure of the NameNode node, and then talk about how the updated NameNode can maintain high availability, even if the crash is down, it can also be quickly restored. In fact, we can think of the strategy of master-slave backup, but it is often easy to say, but complicated to implement. Next, let's take a look.

Let's illustrate it with a picture:

Hadoop1.x:

Hadoop2.x:

The high reliability of HDFS in hadoop2 means that two NameNode can be started at the same time. One of them is working and the other is on standby.

In this way, when the server where a NameNode is located goes down, you can manually or automatically switch to another NameNode to provide services without losing data. These NameNode share data to ensure that the state of the data is consistent. Data is shared among multiple NameNode, either through Nnetwork File System or Quorum Journal Node. The former is a file system shared through linux, which belongs to the configuration of the operating system, while the latter belongs to the hadoop itself and belongs to the configuration of the software. Here we describe how to configure using Quorum Journal Node. You can make it manual or you can configure it automatically (just use zookeeper).

As for the configuration method of NFS, visit: http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html

Look at the documents on apache than I do, the above information is more comprehensive. I don't even want to write a blog!

When the cluster starts, you can start two NameNode at the same time. Only one of these NameNode is active, and the other belongs to the standby state.

The active state means providing the service, and the standby state means being dormant, only synchronizing the data, and being ready to provide the service at all times.

Architecture of HA in 2.Hadoop2

In a typical HA cluster, each NameNode is a separate server. At any one time, only one NameNode is in the active state and the other is in the standby state. Among them, the NameNode in the active state is responsible for all the client operations, and the NameNode in the standby state is in a subordinate position, maintaining the data state and ready to switch at any time.

In order to synchronize data, two NameNode communicate with each other through a set of independent processes called JournalNodes. When there is any change in the namespace of the NameNode in the active state, most JournalNodes processes are notified. NameNode in standby state has the ability to read changes in JNs, and keeps monitoring changes in edit log, applying changes to its own namespace. Standby ensures that the namespace state is fully synchronized when the cluster goes wrong, as shown in the figure:

To ensure fast switching, it is necessary for the NameNode in the standby state to know the location of all data blocks in the cluster. To do this, all datanodes must configure two NameNode addresses and send block location information and heartbeats to both of them.

For HA clusters, it is critical to ensure that only one NameNode is in the active state at a time. Otherwise, the data state of the two NameNode will diverge and may lose data or produce incorrect results. To ensure this, JNS must ensure that only one NameNode can write data to itself at a time.

Note: in a HA cluster, NameNode in standby state can complete checkpoint operations, so there is no need to configure Secondary NameNode, CheckpointNode, and BackupNode. If it is really configured, it will also report an error.

3. Automatic failover

There is a foreign-style name called automaticaly failover.

The manual failover configuration is called configure a manual failover. In other words, in manual mode, the system does not automatically initiate failover, which is the process of Active---- "Stanby. You must execute a command: hdfs haadmin-failover-- forcefence serviceId serviced2. The state will be safely switched. Among them, serviceId2 will become active.

But automatic fault handling is not so simple. Automatic fault handling requires two components: a ZooKeeper quorum and ZKFailoverController process (ZKFC). Among them, ZooKeeper mainly does two things:

Failure detection failure detection, the persistent session of each NameNode machine in the cluster is saved in ZooKeeper management. If the machine crashes, zookeeper will expire the session and notify that another failover will be triggered.

For the election of Active NameNode election:Active NameNode, zookeeper provides a simple mechanism to specifically select a node as active. If the currently active NameNode crashes, another node may get a special mutex lock to indicate that it should be the next active NameNode.

ZKFailoverController is actually a client of zookeeper, which monitors and manages the status of NameNode at the same time. Every NameNode running ZKFC has the following functions:

Health monitoring, ZooKeeper session management, Zookeeper basic election.

Machine allocation of 4.Hadoop Cluster

Hardware: 7 CentOS virtual machines, 512 memory, 20 GB hard disk.

Network:

Host IP installed software yun10-1192.168.2.11jdk, hadoopyun10-2192.168.2.12jdk, hadoopyun10-3192.168.2.13jdk, hadoopyun10-4192.168.2.14jdk, hadoopyun10-5192.168.2.15jdk, hadoop, zookeeperyun10-6192.168.2.16jdk, hadoop, zookeeperyun10-7192.168.2.17jdk, hadoop, zookeeper

Services:

Yun10-1namenodezkfc

Yun10-2namenodezkfc

Yun10-3

Resourcemanageryun10-4

Resourcemanageryun10-5datanodejournalNodeQuorumPeerMainnodemanageryun10-6datanodejournalNodeQuorumPeerMainnodemanageryun10-7datanodejournalNodeQuorumPeerMainnodemanager

Description:

1. In hadoop2.0, it usually consists of two NameNode, one in the active state and the other in the standby state. Active NameNode provides external services, while Standby NameNode does not provide external services, but only synchronizes the status of active namenode so that it can quickly switch if it fails. Hadoop2.0 officially provides two HDFS HA solutions, one is NFS, the other is QJM. Here we use a simple QJM. In this scheme, the master / slave NameNode synchronizes metadata information through a set of JournalNode. As long as a piece of data is successfully written to most JournalNode, the write is considered successful. Usually configure an odd number of JournalNode. A zookeeper cluster is also configured for ZKFC (DFSZKFailoverCont roller) failover. When the Active NameNode is down, the Standby NameNode is automatically switched to the standby state.

There is still a problem in 2.hadoop-2.2.0, that is, there is only one ResourceManager, and there is a single point of failure. Hadoop-2.4.1 solves this problem. There are two ResourceManager, one is Active, the other is Standby, and the status is coordinated by zookeeper.

5. Install and configure zookeeper cluster

Omit. See link: http://my.oschina.net/codeWatching/blog/367309

6. Install and configure Hadoop cluster

The configuration files for Hadoop are centralized under / etc/hadoop/

a. Modify hadoop-env.sh

Export JAVA_HOME=/home/lisai/app/jdk1.7.0_55

b. Modify core-site.xml

c. Modify hdfs-site.xml

d. Modify mapred-site.xml

e. Modify yarn-site.xml

f. Modify slaves (slaves specifies the location of the child node, because you want to start HDFS on yun10-1 and yarn on yun10-3, the slaves file on yun10-1 specifies the location of datanode, and the slaves file on yun10-3 specifies the location of nodemanager)

Yun10-5

Yun10-6

Yun10-7

g. Configure password-free login

# first configure password-free login from yun10-1 to yun10-2, yun10-3, yun10-4, yun10-5, yun10-6, yun10-7

# produce a pair of keys on yun10-1

Ssh-keygen-t rsa

# copy the public key to other nodes, including yourself

Ssh-coyp-id yun10-1

Ssh-coyp-id yun10-2

Ssh-coyp-id yun10-3

Ssh-coyp-id yun10-4

Ssh-coyp-id yun10-5

Ssh-coyp-id yun10-6

Ssh-coyp-id yun10-7

# configure password-free login from yun10-3 to yun10-4, yun10-5, yun10-6, yun10-7

# produce a pair of keys on yun10-3

Ssh-keygen-t rsa

# copy the public key to another node

Ssh-coyp-id yun10-4

Ssh-coyp-id yun10-5

Ssh-coyp-id yun10-6

Ssh-coyp-id yun10-7

# Note: configure ssh password-free login between two namenode. Don't forget to configure yun10-2 to yun10-1 login-free.

Produce a pair of keys on yun10-2

Ssh-keygen-t rsa

Ssh-copy-id-I yun10-1

7. Copy of configuration information scp-r. / hadoop/ yun10-2:/home/lisai/app/hadoop-2.4.1/etc/hadoop/scp-r. / hadoop/ yun10-3:/home/lisai/app/hadoop-2.4.1/etc/hadoop/scp-r. / hadoop/ yun10-4:/home/lisai/app/hadoop-2.4.1/etc/hadoop/scp-r. / hadoop/ yun10-5:/home/lisai/app/hadoop-2.4 .1 / etc/hadoop/scp-r. / hadoop/ yun10-6:/home/lisai/app/hadoop-2.4.1/etc/hadoop/scp-r. / hadoop/ yun10-7:/home/lisai/app/hadoop-2.4.1/etc/hadoop/8. Start the Hadoop cluster

a. Start ZooKeeper.

. / bin/zkServer.sh start./bin/zkserver.sh status

b. Start journalnode (executed on yun10-5, yun10-6, yun10-7, respectively)

. / sbin/hadoop-daemon.sh start journalnodejps

c. Format HDFS

# execute the command on yun10-1: hdfs namenode-format# format will generate a file based on the hadoop.tmp.dir configuration in core-site.xml. Here I configure / home/lisai/app/hadoop-2.4.1/tmp, and then copy / home/lisai/app/hadoop-2.4.1/tmp to yun10-2 under / home/lisai/app/hadoop-2.4.1. Scp-r tmp/ yun10-2:/home/lisai/app/hadoop-2.4.1/tmp

d. Format ZK

# execute hdfs zkfc-formatZK on yun10-1

d. Start HDFS

# execute the NameNode process in yun10-1. / sbin/start-hdfs.sh# executes the NameNode process in yun10-2. / sbin/hadoop-daemon.sh start NameNode/ may be lowercase. # in yun10-5, yun10 10-6, Yun 10-7, others start. / sbin/hadoop-daemon.sh start datanode

e. Start YARN (# Note #: execute start-yarn.sh on yun10-3, separate namenode and resourcemanager because of performance problems, because they both take up a lot of resources, so separate them, they have to start on different machines if they are separated)

# yun10-3./sbin/start-yarn.sh#yun10-4./sbin/yarn-daemon.sh start resourcemanager9. Verify the cluster

HDFS: http://yun10-1:50070

MapReducer: http://yun10-3:8090

The above is all the contents of the article "example Analysis of HA Mechanism and Hadoop Cluster Construction in HDFS". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.