Shulou(Shulou.com)06/01 Report--

This article shows you what the function of reportWrittenBlock in HDFS is. The content is concise and easy to understand. It can definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

/ * *

* The client can report in a set written blocks that it wrote.

* These blocks are reported via the client instead of the datanode

* to prevent weird heartbeat race conditions.

, /

Public void reportWrittenBlock (LocatedBlock lb) throws IOException {

Block b = lb.getBlock (); / / get the completed Block information

DatanodeInfo targets [] = lb.getLocations (); / / get node information

For (int I = 0; I

< targets.length; i++) { namesystem.blockReceived(b, targets[i].getName());//对于每个DataNode来说，都要调用一次此函数 } } C1：2014-12-19 18:26:00 C2:2014-12-19 18:59:00 C3:2014-12-19 19:03:00 ========================= 那么，接下来就是理解 namesystem.blockReceived(b, targets[i].getName());了。 /** * The given node is reporting that it received a certain block. */ public synchronized void blockReceived(Block block, UTF8 name) { DatanodeInfo node = (DatanodeInfo) datanodeMap.get(name);//获取对应的datanode if (node == null) {//为空可不行 throw new IllegalArgumentException("Unexpected exception. Got blockReceived message from node " + name + ", but there is no info for " + name); } // // Modify the blocks->

Datanode map

/ /

AddStoredBlock (block, node); / / the following two lines are used to perform a mapping of block and node.

/ /

/ / Supplement node's blockreport

/ /

Node.addBlock (block); / / Ibid.

}

C1RO 2014-12-19 19:11:00 C2RO 2014-12-19 19:11:00 C3RO 2014-12-19 19:12:00

= then there are two more functions to break, namely addStoredBlock and node.addBlock (block)

The latter function is very simple and won't go into detail, so there is only one function left!

The execution process of addStoredBlock (block, node); is as follows:

Synchronized void addStoredBlock (Block block, DatanodeInfo node) {

TreeSet containingNodes = (TreeSet) blocksMap.get (block); / / get the datanode information that already exists in the current block

If (containingNodes = = null) {/ / it is guaranteed that there must be datanode sets, not nodes.

ContainingNodes = new TreeSet ()

BlocksMap.put (block, containingNodes)

}

If (! ContainingNodes.contains (node) {/ / decide whether or not to join this datanode information as needed

ContainingNodes.add (node)

} else {

LOG.info ("Redundant addStoredBlock request received for block" + block + "on node" + node)

}

/ / the next logic is to determine whether a re-backup is needed.

Synchronized (neededReplications) {/ / Lock neededReplications

If (dir.isValidBlock (block)) {/ / does not understand this sentence.

If (containingNodes.size () > = this.desiredReplication) {/ / if the maximum number of backups has been exceeded

NeededReplications.remove (block); / / Delete this block

} else if (containingNodes.size ()

< this.desiredReplication) { if (! neededReplications.contains(block)) { neededReplications.add(block);//否则表示需要重新备份，这代码写的真够差的。。。 } } // // Find how many of the containing nodes are "extra", if any. // If there are any extras, call chooseExcessReplicates() to // mark them in the excessReplicateMap. // //也有可能一个block存储的datanode节点数太多了，同样要删除这些block Vector nonExcess = new Vector();//构造一个空的Vector for (Iterator it = containingNodes.iterator(); it.hasNext(); ) { DatanodeInfo cur = (DatanodeInfo) it.next();//对于当前节点来说 TreeSet excessBlocks = (TreeSet) excessReplicateMap.get(cur.getName());//取到当前节点的多余块信息 if (excessBlocks == null || ! excessBlocks.contains(block)) {//如果之前没有标志在这个节点的多余块信息里 nonExcess.add(cur);//则表明当前节点存储了这个block } } if (nonExcess.size() >

ChooseExcessReplicates (nonExcess, block, this.maxReplication); / / Select several to eliminate blocks

}

}

}

}

Int chosenNode = r.nextInt (nonExcess.size ()); / / randomly select a node

DatanodeInfo cur = (DatanodeInfo) nonExcess.elementAt (chosenNode)

NonExcess.removeElementAt (chosenNode); / / get this node

TreeSet excessBlocks = (TreeSet) excessReplicateMap.get (cur.getName ())

If (excessBlocks = = null) {

ExcessBlocks = new TreeSet ()

ExcessReplicateMap.put (cur.getName (), excessBlocks)

}

ExcessBlocks.add (b); / / add this block to excessReplicateMap

/ /

/ / The 'excessblocks' tracks blocks until we get confirmation

/ / that the datanode has deleted them; the only way we remove them

/ / is when we get a "removeBlock" message.

/ /

/ / The 'invalidate' list is used to inform the datanode the block

/ / should be deleted. Items are removed from the invalidate list

/ / upon giving instructions to the namenode.

/ /

Vector invalidateSet = (Vector) recentInvalidateSets.get (cur.getName ())

If (invalidateSet = = null) {

InvalidateSet = new Vector ()

RecentInvalidateSets.put (cur.getName (), invalidateSet)

}

InvalidateSet.add (b); / / again, update recentInvalidateSets, there is nothing to explain

}

}

The above is what is the function of reportWrittenBlock function in HDFS. Have you learned any knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.