Shulou(Shulou.com)06/01 Report--

This article mainly explains the "introduction to the startup process of DataNode in Hadoop". The content in the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "introduction to the startup process of DataNode in Hadoop".

DataNode, like NameNode, is a java process, so start with the main method and look at the code:

Public class DataNode extends Configured implements InterDatanodeProtocol, ClientDatanodeProtocol, DataNodeMXBean {public static void main (String args []) {secureMain (args, null);} public static void secureMain (String args [], SecureResources resources) {DataNode datanode = createDataNode (args, null, resources);} public static DataNode createDataNode (String args [], Configuration conf, SecureResources resources) throws IOException {DataNode dn = instantiateDataNode (args, conf, resources); if (dn! = null) {dn.runDatanodeDaemon () } return dn;} public static DataNode instantiateDataNode (String args [], Configuration conf, SecureResources resources) throws IOException {return makeInstance (dataLocations, conf, resources);} static DataNode makeInstance (Collection dataDirs, Configuration conf, SecureResources resources) throws IOException {return new DataNode (conf, locations, resources);} DataNode (final Configuration conf, final List dataDirs, final SecureResources resources) throws IOException {startDataNode (conf, dataDirs, resources) } public void runDatanodeDaemon () throws IOException {blockPoolManager.startAll (); / / start dataXceiveServer dataXceiverServer.start (); if (localDataXceiverServer! = null) {localDataXceiverServer.start ();} ipcServer.start (); startPlugins (conf);}}

The above code tracking is not difficult, but in the end, you need to pay attention to two methods: the startDataNode and runDatanodeDaemon methods, the former is used to initialize the DataNode, and the latter starts the background thread of DataNode, which runs all the time with the DataNode process. Next, let's focus on the method startDataNode and look at the code:

Void startDataNode (Configuration conf, List dataDirs, / / DatanodeProtocol namenode, SecureResources resources) throws IOException {storage = new DataStorage (); / / global DN settings registerMXBean (); initDataXceiver (conf); startInfoServer (conf); pauseMonitor = new JvmPauseMonitor (conf); pauseMonitor.start (); initIpcServer (conf) BlockPoolManager = new BlockPoolManager (this); blockPoolManager.refreshNamenodes (conf);}

The registerMXBean method can be ignored and is used to register MBean information; the initDataXceiver method should still be important. The instantiated dataXceiverServer is used to accept data receiving or sending requests from the client or other datanode; the startInfoServer method is used to start datanode's web server; pauseMonitor is used to monitor whether the jvm is paused; and the initIpcServer method is more important, which is used to start rpc services on datanode, including two services: ClientDatanodeProtocolPB and InterDatanodeProtocolPB.

Then comes the focus of DataNode, the instantiation of blockPoolManager objects. Note that the 2.4.1 version of hadoop already supports the hadoop Federation feature, and blockPooolManager supports this feature. Now let's look at what's inside him. Let's code it first.

Class BlockPoolManager {BlockPoolManager (DataNode dn) {this.dn = dn;} void refreshNamenodes (Configuration conf) throws IOException {synchronized (refreshNamenodesLock) {doRefreshNamenodes (newAddressMap);}} private void doRefreshNamenodes (Map addrMap) throws IOException {synchronized (this) {startAll ();}} synchronized void startAll () throws IOException {for (BPOfferService bpos: offerServices) {bpos.start () } class BPOfferService {void start () {for (BPServiceActor actor: bpServices) {actor.start ();} class BPServiceActor implements Runnable {void start () {if ((bpThread! = null) & & (bpThread.isAlive () {/ / Thread is started already return;} bpThread = new Thread (this, formatThreadName ()); bpThread.setDaemon (true); / / needed for JUnit testing bpThread.start () } public void run () {while (true) {connectToNNAndHandshake (); break;} while (shouldRun ()) {offerService ();}} private void offerService () throws Exception {while (shouldRun ()) {HeartbeatResponse resp = sendHeartBeat (); List cmds = blockReport ();}

As you go down the code, the whole idea becomes clearer. The BPServiceActor class does specific things, including shaking hands between datanode and namenode, sending heartbeats and report block messages, and performing the naming sent back by namenode.

The detailed process will not be verbose.

At this point, there is a paragraph in the startup process of DataNode.

Thank you for your reading, the above is the content of "introduction to the startup process of DataNode in Hadoop". After the study of this article, I believe you have a deeper understanding of the introduction of the startup process of DataNode in Hadoop, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.