The difference between Hadoop2.x and hadoop

When NameNode starts, the Mettadata needs to be loaded into memory. As the cluster expands, the amount of metadata increases, and the memory pressure is too high.

Hadoop1.x 's shortcomings and solutions:

Single point of failure-à HA with one master and multiple backup

Memory constraint-à NN Federation

The communication between Hadoop nodes is based on heartbeat mechanism.

The metadata is placed in a shared area (the primary NN and the standby NN are shared) and is called a Journal Node cluster

All DN nodes communicate with the primary NN or standby NN in real time, telling NN what information they have, such as block.

As for whether DN should ask NN (master) or NN (standby), it will ask zookeeper for instructions. Zookeer knows which node is the master and knows its status.

When the state of the main NN is detected by the FailoverControllerActive, the zookeeper is notified, and when it fails, the standby NN also has the corresponding FailoverControllerStandby.

Through the command, you can manually stop the master NN to start the standby NN, or even change roles for HDFS upgrade and other occasions.

NN Fderation: split the HDFS into multiple completely independent NN operations that do not interfere with each other, but the cluster shares the DN

Multiple NN share the storage resources on the DN in a cluster, and each NN can provide services separately.

Each NN defines a storage pool with a separate id, and each DN provides storage for all storage pools

DN reports block information to its corresponding NN according to the storage pool id, and DN reports the resources available for local storage to all NN

If you need convenient access to resources on several NN on the client, you can use the client mount table to map different directories to different NN, but the corresponding directories must exist on the NN

Main functions of Yarn:

Responsible for resource management, task scheduling

Support a variety of third-party computing frameworks