Shulou(Shulou.com)06/03 Report--

This article mainly explains "how to understand HA mechanism of Namenode". The explanation content in this article is simple and clear, easy to learn and understand. Please follow the ideas of Xiaobian and go deep into it slowly to study and learn "how to understand HA mechanism of Namenode" together.

[Overall design]

The HA mechanism of NN (Namenode) mainly relies on zkfc, which runs as an independent process in the node where NN is located. It is mainly realized by three modules: ZK Failover Controller, Health Monitor and Active Stand by Elector.

When zkfc process starts, ZKFailoverController will be created first, which is responsible for handling callback events of the remaining two modules.

Health detection module sends rpc request to NN periodically to monitor health status.

The master-slave election module encapsulates the processing logic of zookeeper, including the establishment of tcp connection, the creation of nodes, the change of watch nodes, etc.

When the health status of NN changes, the health detection module will call back to notify the main control module, and then trigger the election module to vote or withdraw from the election.

Similarly, when the election module detects a change in the state of a node on the zookeeper, it will automatically trigger an election, then call back to notify the main control module, and finally notify NN to become active or standby through the rpc.

Zkfc's start-up election process is shown below:

After zkfc starts, first construct the election module and establish a connection to zookeeper.

Then start the health detection of NN, send rpc request to NN, and obtain the state and health of NN.

If NN is healthy, an election is triggered, i.e. temporary lock nodes are created on zookeeper.

Zk's own mechanism ensures that only one zkfc can successfully create a node.

For zkfc that successfully creates a node, it sends an rpc request to other NNs for fencing (essentially telling the peer to become standby), then creates a persistent node (ActiveBreadCrumb) on zookeeper that records NN master-slave information, and finally sends an rpc request to the local NN to inform it to become ANN (ActiveNameNode).

Zkfc, which failed to create a node, sends an rpc request to the local NN through callback, telling it to become standby, and then watches the lock node created by ANN in zookeeper.

[HA Switching Scenes]

ANN anomaly

When ANN is abnormal (including process exit, status and health of RPC request no response, etc.), zkfc will actively withdraw from the election, that is, end the TCP connection with zookeeper, the corresponding session of the connection will be automatically deleted on the zookeeper Lock node created.

Watch (running on SNN) zkfc on this node perceives the change of node, triggers election (re-creates lock node on zookeeper), then obtains the value of ActiveBreadCrumb node, obtains the information of old ANN node from it, when it is found that the old ANN is not this node, zkfc triggers fencing on it, then writes new information on ActiveBreadCrumb node, and finally notifies SNN to become new ANN.

zkfc anomaly &ANN node anomaly

The difference between these two scenarios and the above process is that zookeeper detects the timeout of the connection established by zkfc and automatically deletes the temporary node created by the session on this connection. The next step is the same as above.

[Notes]

Handling of fencing

Both the election process and HA handover process mentioned fencing (isolation), so why is it good to carry out fencing? What is the significance of fencing?

For the above zkfc abnormal scene, let's analyze it in depth:

When the zkfc of the node where the ANN is located is abnormal, or only the network instability between zkfc and zookeeper causes the session timeout between zkfc and zookeeper, thus triggering the zkfc election on the snn node and becoming the new ANN.

If no fencing is performed, then there are two ANNs at this time, and the service is provided to the outside at the same time, which may cause the hdfs data to be inconsistent or even corrupted and cannot be recovered.

Therefore, SNN needs to perform fencing on old ANN before it becomes new ANN.

Specifically, zkfc sends an rpc request directly to the old ANN informing it that it is an SNN, a process known as elegant fencing.

If the old ANN responds successfully, zkfc then informs the SNN to become the new ANN.

If the old ANN does not respond, fencing is performed again depending on how configured.

Configurable modes include ssh and execution of specified scripts.

The ssh method is zkfc to pass ssh to the old ANN node, then execute the kill action to kill the old ANN, and finally notify the SNN to become the new ANN. If the ANN node is abnormal, ssh will not succeed, so the old ANN cannot be killed.

Therefore, the usual way is to execute your own script and perform the relevant processing actions in the script. zkfc returns a decision to notify SNN to become a new ANN through script, or triggers the election process again.

Definition of health status

As mentioned above, the health detection module sends rpc requests to NN regularly to obtain nn's status (Active/Standby/Initializing) and health status. How does NN determine whether it is healthy?

Tracking its source code found: NN configured local directory (used to store fsimage and editlog directory and other specified configuration directory) disk capacity check, if the directory corresponding to the disk capacity reached the minimum configured value, NN considered itself healthy, otherwise considered unhealthy.

Make sure parent exists

After zkfc is started and successfully connected to zookeeper, it will first store whether the parent node of the lock node exists. If it does not exist, the zkfc process will exit directly.

When zkfc starts, you can specify the format parameter, in which case zkfc deletes the information stored on zookeeper, creates the required parent node, and then the process exits. When started again (with no parameters), the election will proceed correctly.

In addition, if the corresponding parent node is deleted during the operation of zkfc, zkfc will not be automatically created again. In this case, the lock node for election will continue to fail to be created, resulting in the failure of normal election, resulting in NN unable to provide services normally.

Ensure NN ID matches IP

After successful NN election, ANN will create ActiveBreadCrumb node on zookeeper to record ANN/SNN ID and corresponding IP information.

Because the node is persistent, when the NN restarts or re-elects, it reads the value of the node and obtains the IP information of the old ANN from it for fencing processing. Before this, the NN ID recorded by the node will be compared with the corresponding IP and local configuration. If it is inconsistent with the configuration, an exception will be thrown and no subsequent processing will be performed.

This usually happens when NN is deployed and run as a container. When the container where NN is located goes offline and restarts, the IP assigned by NN changes, resulting in inconsistency with the information recorded in ActiveBreadCrumb node in zookeeper.

Thank you for reading, the above is "how to understand Namenode HA mechanism" content, after the study of this article, I believe that everyone on how to understand Namenode HA mechanism this problem has a deeper experience, the specific use of the situation also needs to be verified by practice. Here is, Xiaobian will push more articles related to knowledge points for everyone, welcome to pay attention!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.