Shulou(Shulou.com)06/03 Report--

This article mainly explains "how DataNode sends heartbeats to NameNode". The explanation in the article is simple and clear, easy to learn and understand. Please follow the editor's train of thought to study and learn "how DataNode sends heartbeats to NameNode".

1. Heartbeat technology in distributed system

Heartbeat is a commonly used technical means in distributed technology. Heartbeat, as the name implies, is the way to report the current node status to other nodes at a fixed frequency. When a heartbeat is received, it is generally considered that the node that sends the heartbeat is good in the current network topology.

Of course, when the heartbeat reports, it will generally carry some additional status, metadata information, current node information, so that other nodes can manage.

There are two ways to beat the heart:

Unidirectional heartbeat

Interactive ping-pong

In the first way, the target process needs to send messages to detector periodically to inform it of its viability. On the other hand, detector does not need to reply any messages to target, but only checks whether the target process has been reported from time to time.

The second method is more common. For example, our redis uses this method:

Detector-> target: Are you ok? Target-> detector: Yeah, pretty good.

The Detector initiates the detection, and if the target does not reply to the message for N consecutive times, then detector considers it to be in the non-active state.

So what are the common heartbeat detection mechanisms:

(1) traditional periodic heartbeat detection mechanism.

The detection method is very rough: set a timeout T, as long as the other party's heartbeat packet is not received within T, the other party can be considered to be down, the method is simple and effective, and it is widely used.

Therefore, the focus of this method is that the setting of the timeout time T is too short, it may lead to miscalculation because of the current network congestion, and let this node go offline, resulting in other unnecessary consequences; if it is set too long, it will lead to "slow" judgment, so it needs to be weighed and set in a variety of situations.

HDFS uses this heartbeat mechanism.

(2) cumulative failure detection mechanism

With the increase of network load, the receiving time of Server heartbeat may be greater than the upper limit value T, but when the network pressure decreases, the heartbeat receiving time will be less than T. if the constant T is used to reflect the heartbeat condition, it will cause "delay" or misjudgment. At this time, we can calculate the probability of heartbeat delay, and use this probability to judge whether there is a failure and improve the accuracy.

2. How DataNode sends heartbeats to NameNode

Let's see how DataNode sends heartbeats from the hadoop source code.

1. Start with the main method of the DataNode class

2. Create DataNode

3. Instantiate DataNode

4. Create an instance

5. New DataNode

6. The constructor of this method is a little long and pulled to the bottom

7. Then come to this method

This method appears to refresh the NameNode, but in fact it does two things to register itself with the NameNode, and the other thing is to send a regular heartbeat to the NameNode.

Click in, now it is in BlockPoolManager

9. Come back to this method

10. Find it traversing the BPOfferService

Here is a rough description of what the data structure offerServices is.

Show it with a picture.

That offerServices is actually the leftmost BPOfferService.

There are two BpServiceActor in each BPOfferService, and one NameNode for each BpServiceActor.

If it is the high availability in the figure above, then one master and one slave have two NameNode, each corresponding to one BpServiceActor.

So traversing offerService is actually traversing every NameNode node of every federation in the entire cluster.

11. Start traversing (current class: BpOfferService)

12. Thread start (current class: BPServiceActor)

Clicking on it is already the method of Thread, which shows that it is actually a thread.

Then we should look at the run method of this class.

13. Running content of the thread

The top part of the run method registers with NameNode, and the bottom part sends a heartbeat.

14. Send a heartbeat

You can see that this is a while loop that is executed at regular intervals (dnConf.heartBeatInterval = 3 seconds)

You can see that the heartbeat of the DataNode uses a periodic detection mechanism to send heartbeats to all NameNode every 3 seconds.

15. Click in again, it is already the proxy proxy method of NameNode.

Because it communicates through rpc, DataNode is the client and NameNode is the server.

At this point, we should look at the NameNodeRpcServer method, the specific implementation is in this method.

16. Look at the processing on the server side. At this time, we are in the NameNodeRpcServer class

I don't want to look at the details of this method (in fact, it's also very important).

Basically, the basic information of the DataNode heartbeat packet (such as the storage capacity of this node, etc.) is updated to the corresponding structure of the NameNode.

And update the last heartbeat time so that the next time you can determine if DataNode's heartbeat is timed out.

In fact, the heartbeat is so simple.

Finally, NameNode will tell DataNode what to do in the heartbeat response, such as backing up the Block of this node to other nodes.

In other words, NameNode itself does not communicate with DataNode, but instead tells DataNode what to do in the heartbeat message.

Thank you for your reading, the above is the content of "how DataNode sends heartbeat to NameNode". After the study of this article, I believe you have a deeper understanding of how DataNode sends heartbeat to NameNode, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.