Shulou(Shulou.com)06/02 Report--

This article will explain in detail how to use Hadoop source code to analyze the heartbeat mechanism, the content of the article is of high quality, so the editor will share it with you for reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

I. heartbeat mechanism

1. Hadoop cluster is in master/slave mode. Master includes Namenode and Jobtracker,slave, including Datanode and Tasktracker.

2. When master starts, an ipc server will be opened there, waiting for the heartbeat of slave.

3. When slave starts, it connects to master and actively sends a "heartbeat" to master every 3 seconds, which can be set through the "heartbeat.recheck.interval" property. Tell master its status information, and master also sends instructions to the slave node through the return value of this heartbeat.

4. It should be pointed out that the communication between namenode and datanode and between jobtracker and tasktracker is accomplished through the heartbeat.

II. Analysis of heartbeat source code of Datanode and Namenode

Since the heartbeat was sent to Namenode by Datanode. So how is Datanode sent? The key code in Datanode.class is posted below:

Code 1:

/ * call the "send heartbeat" method in a loop until shutdown * calls the remote Namenode method * / public void offerService () throws Exception {while (shouldRun) {try {long startTime = now () / / heartBeatInterval is set according to the configuration file when starting Datanode, and is the heartbeat interval if (startTime-lastHeartbeat > heartBeatInterval) {lastHeartbeat = startTime / / Datanode sends heartbeat DatanodeCommand [] cmds = namenode.sendHeartbeat (dnRegistration, data.getCapacity (), data.getDfsUsed (), data.getRemaining ()) XmitsInProgress.get (), getXceiverCount () MyMetrics.addHeartBeat (now ()-startTime); if (! processCommand (cmds)) continue;}} / / while (shouldRun)} / / offerService

It should be noted that the object that sends the heartbeat is not datanode, but an object called namenode. Is there a reference to namenode directly on the datanode side? Actually, let's take a look at this namenode:

Code 2:

Public DatanodeProtocol namenode = null

Namenode is actually a reference to DatanodeProtocol, which I mentioned in my analysis of the hadoop RPC mechanism is a protocol for communicating between Datanode and Namenode, and sendHeartbeat () is one of many unimplemented interface methods. Let's see how this namenode object is instantiated:

Code 3:

This.namenode = (DatanodeProtocol) RPC.waitForProxy (DatanodeProtocol.class, DatanodeProtocol.versionID, nameNodeAddr, conf)

In fact, this namenode is not an object of Namenode, but just a Datanode end-to-Namenode proxy object, which completes the heartbeat. The underlying implementation of the agent is the RPC mechanism.

III. Tasktracker, Jobtracker heartbeat source code analysis

Similarly, let's start with Tasktracker, and the key code of Tasktracker.class is posted below:

Code 4:

Code 1: State offerService () throws Exception {long lastHeartbeat = System.currentTimeMillis (); while (running & &! shuttingDown) {/ / send heartbeat, call code 2 HeartbeatResponse heartbeatResponse = transmitHeartBeat (now); return State.NORMAL } Code 2: HeartbeatResponse transmitHeartBeat (long now) throws IOException {HeartbeatResponse heartbeatResponse = jobClient.heartbeat (status, justStarted, justInited) AskForNewTask, heartbeatResponseId) Return heartbeatResponse;}

In fact, I think you can analyze it yourself at this point. JobClient is also an agreement:

Code 5:

InterTrackerProtocol jobClient

This protocol is used to define communication between Tasktracker and Jobtracker. Again, it is a proxy object:

Code 6:

This.jobClient = (InterTrackerProtocol) UserGroupInformation.getLoginUser () .doAs (new PrivilegedExceptionAction () {public Object run () throws IOException {return RPC.waitForProxy (InterTrackerProtocol.class, InterTrackerProtocol.versionID, jobTrackAddr, fConf);}})

Finally, the source code analysis of the whole series of hadoop underlying communications has been completed.

On how to use Hadoop source code analysis heartbeat mechanism to share here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.