Shulou(Shulou.com)06/01 Report--

What is the role of the create function in HDFS, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain in detail for you, people with this need can come to learn, I hope you can get something.

After client learned that this file does not exist on the current namenode side through the exists () function

A file is created through the namenode.create function. The details are as follows:

This means that the clientName of clientMachine creates the src file.

ClientMachine is only used to select the target DataNode.

Public LocatedBlock create (String src, String clientName, String clientMachine, boolean overwrite) throws IOException {

Object results [] = namesystem.startFile (new UTF8 (src), new UTF8 (clientName), new UTF8 (clientMachine), overwrite); / / call the startFile function of the file system, and the returned values are block information and target datanode information

If (results = = null) {

Throw new IOException ("Cannot create file" + src + "on client" + clientName)

} else {

Block b = (Block) results [0]; / / retrieve block

DatanodeInfo targets [] = (DatanodeInfo []) results [1]; / / get DatanodeInfo array information

Return new LocatedBlock (b, targets); / / the combination returns the final information

}

}

= =

Let's start to learn.

Public synchronized Object [] startFile (UTF8 src, UTF8 holder, UTF8 clientMachine, boolean overwrite) {

The analysis of this function is as follows:

Public synchronized Object [] startFile (UTF8 src, UTF8 holder, UTF8 clientMachine, boolean overwrite) {

/ / background knowledge: parameters include holder and clientMachine. For example, an example is as follows:

Holder:DFS_CLIENT_xxxx

ClientMachine:Machine66.

That is to say, a clientMachine can have more than one Holder.

A Holder on clientMachine sent a request for upload.

Where holder is used and where clientMachine is used in the following code

Readers are also asked to pay attention to their own thinking.

Object results [] = null

If (pendingCreates.get (src) = = null) {/ / indicates that pendingCreates records the file being created

Boolean fileValid = dir.isValidToCreate (src); / / the file path really doesn't exist, do you need this sentence?

If (overwrite &! FileValid) {/ / if it can be overwritten, it cannot be overwritten at present.

Delete (src)

FileValid = true

}

If (fileValid) {/ / if you can, continue to execute

Results = new Object [2]; / / create an array that returns results

/ / Get the array of replication targets

DatanodeInfo targets [] = chooseTargets (this.desiredReplication, null, clientMachine)

/ / Select multiple destination datanode based on clientMachine and number of backups

If (targets.length

< this.minReplication) { LOG.warning("Target-length is " + targets.length + ", below MIN_REPLICATION (" + this.minReplication+ ")"); return null; }//如果长度达不到备份数，则返回失败 // Reserve space for this pending file pendingCreates.put(src, new Vector());//表明这个文件正在create!!! synchronized (leases) {//开始处理租约系统 Lease lease = (Lease) leases.get(holder);//查找租约系统 if (lease == null) {//如果不存在 lease = new Lease(holder);//创建 leases.put(holder, lease);//存储到leases sortedLeases.add(lease);//存储到sortedLeases } else {//如果存在的话，则lease本身刷新时间且重新加入到sortedLeases. //注意，这里有一个sort过程。 sortedLeases.remove(lease); lease.renew(); sortedLeases.add(lease); } lease.startedCreate(src);//lease的本身creates保存了文件名 } // Create next block results[0] = allocateBlock(src);//主要是记录文件对应的Block信息 results[1] = targets;//分配的datanode信息 } else { // ! fileValid LOG.warning("Cannot start file because it is invalid. src=" + src); } } else { LOG.warning("Cannot start file because pendingCreates is non-null. src=" + src); } return results;//返回结果！ } ------------------------------------------------------------- DatanodeInfo[] chooseTargets(int desiredReplicates, TreeSet forbiddenNodes, UTF8 clientMachine) { TreeSet alreadyChosen = new TreeSet();//初始化空的已经选择的机器 Vector targets = new Vector();//真的无语。这里为啥还要再创建一个targets,浪费内存，直接传到chooseTarget一样的好吧！崩溃！ for (int i = 0; i < desiredReplicates; i++) {//根据备份数来选择执行次数 DatanodeInfo target = chooseTarget(forbiddenNodes, alreadyChosen, clientMachine);//选择单个机器 if (target != null) {//选择好了一个，就加到targets和alreadyChosen.崩溃，加2次有啥意思！！！ targets.add(target); alreadyChosen.add(target); } else { break; // calling chooseTarget again won't help } } return (DatanodeInfo[]) targets.toArray(new DatanodeInfo[targets.size()]);//返回执行的结果 } --------------- ======================= DatanodeInfo chooseTarget(TreeSet forbidden1, TreeSet forbidden2, UTF8 clientMachine) { // // Check if there are any available targets at all // int totalMachines = datanodeMap.size();//获取当前已知的所有数据节点个数 if (totalMachines == 0) {//为0就不用说了，返回null LOG.warning("While choosing target, totalMachines is " + totalMachines); return null; } // // Build a map of forbidden hostnames from the two forbidden sets. // TreeSet forbiddenMachines = new TreeSet(); if (forbidden1 != null) {//这里forbidden1是初始化禁止的节点，此处为null for (Iterator it = forbidden1.iterator(); it.hasNext(); ) { DatanodeInfo cur = (DatanodeInfo) it.next(); forbiddenMachines.add(cur.getHost()); } } if (forbidden2 != null) {//是已经选择的节点，因为已经选择的就不会再返回了，你懂的 for (Iterator it = forbidden2.iterator(); it.hasNext(); ) { DatanodeInfo cur = (DatanodeInfo) it.next(); forbiddenMachines.add(cur.getHost()); } } // // Build list of machines we can actually choose from // Vector targetList = new Vector();//从总的节点中去掉不可以选择的节点，得到剩下的可选的节点 for (Iterator it = datanodeMap.values().iterator(); it.hasNext(); ) { DatanodeInfo node = (DatanodeInfo) it.next(); if (! forbiddenMachines.contains(node.getHost())) { targetList.add(node); } } Collections.shuffle(targetList);//本来不知道干嘛的，百度了一下，用来洗牌的 //为啥？因为DFSShell采用计算机组成原理的菊花链的方式来上传数据。剩下的我就不用解释了 // // Now pick one // if (targetList.size() >

0) {/ / if there are really selectable nodes left, and clientMachine is also in it

/ / and if the capacity is greater than 5, return clientMachine directly. I guess it's for local acceleration.

/ / after all, uploading to a local host is not the same as uploading to a remote host.

/ / If the requester's machine is in the targetList

/ / and it's got the capacity, pick it.

/ /

If (clientMachine! = null & & clientMachine.getLength () > 0) {

For (Iterator it = targetList.iterator (); it.hasNext ();) {

DatanodeInfo node = (DatanodeInfo) it.next ()

If (clientMachine.equals (node.getHost () {

If (node.getRemaining () > BLOCK_SIZE * MIN_BLOCKS_FOR_WRITE) {

Return node

}

}

}

}

/ /

/ / Otherwise, choose node according to target capacity

/ / otherwise, select a node with a capacity greater than 5 blocks

For (Iterator it = targetList.iterator (); it.hasNext ();) {

DatanodeInfo node = (DatanodeInfo) it.next ()

If (node.getRemaining () > BLOCK_SIZE * MIN_BLOCKS_FOR_WRITE) {

Return node

}

}

/ /

/ / That should do the trick. But we might not be able

/ / to pick any node if the target was out of bytes. As

/ / a last resort, pick the first valid one we can find.

/ / otherwise, select a node that is at least greater than 1 block

For (Iterator it = targetList.iterator (); it.hasNext ();) {

DatanodeInfo node = (DatanodeInfo) it.next ()

If (node.getRemaining () > BLOCK_SIZE) {

Return node

}

}

LOG.warning ("Could not find any nodes with sufficient capacity")

Return null;// otherwise returns null

} else {

LOG.warning ("Zero targets found, forbidden1.size=" +

(forbidden1! = null? Forbidden1.size (): 0) +

"forbidden2.size () =" +

(forbidden2! = null? Forbidden2.size (): 0))

Return null;// doesn't have a single node to find!

}

}

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.