Shulou(Shulou.com)06/03 Report--

Big Data Learning Routes Share Hadoop Phase High Availability Configuration, What is Hadoop HA Mechanism

Ha mechanism is high availability of Hadoop (7*24 hours uninterrupted service)

HA was formally introduced from hadoop 2.0, and there was no HA mechanism in previous versions.

hadoop-ha Strictly speaking, HA mechanism should be divided into various components-HA of HDFS, HA of YARN

Detailed explanation of HA mechanism of HDFS

HA of HDFS is mainly realized through double namenode coordination

Key points of double namenode coordination work:

A. Metadata management needs to change:

Each of them stores one copy of metadata in memory. Edits log can only have one copy. Only Active namenode nodes can write. Both namenode nodes can read edits. Shared edits are managed in a shared storage.(qjournal and NFS are two mainstream implementations) B. A state management function module is needed to implement a zkfailover, which resides in the node where each namenode is located. Each zkfailover is responsible for monitoring its own namenode node, and uses zk for state identification. When state switching is required, Zkfailover is responsible for switching. When switching, it is necessary to prevent the occurrence of brain split phenomenon.

How Namenode works

There is a namenode on both servers. One namenode is active and the other is standby. The two servers share data. Each server has one copy of metadata, but only one copy of edit data. Only the namenode server in active state can write edit, and the other server can only read edit. The shared edit is managed in a shared storage. Shared storage is implemented by file management systems qjournal and NFS.

How to manage the active standby state of the two servers requires a management module: ZKFC (zookeeper failover controller). Each zkfc is responsible for monitoring its own namenode node, using zk for state identification. When state switching is required, zkfailover is responsible for switching.

When switching, it is necessary to prevent the occurrence of brain split phenomenon.

What is split-brain phenomenon?

Split brain phenomenon is that two namenode are in active state, resulting in conflict, this is split brain. Hadoop's high availability configuration should pay attention to solving the split brain state.

How does the split-brain state arise?

When an active namenode server is suspended, the zkfc of another namenode server receives a message to change its namenode status to active, and the first namenode in suspended animation wakes up again, resulting in brain splitting.

How to solve brain split

The zkfc of the second namenode will not stop at this time, killing the namenode in the first fake death state using ssh kill -9 namenode , directly killing the namenode of the first server to make up the knife, if the knife is not successful, zkfc enters the first server and directly calls the user's custom script program/home/Hadoop/kill/poweroff.sh to kill-kill fake-dead namenode.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.