Shulou(Shulou.com)06/01 Report--

Xiaobian to share with you how Hadoop 2.6 cluster dynamically add and delete data nodes, I believe most people do not know how, so share this article for your reference, I hope you read this article after a great harvest, let us go to understand it!

Before you start, you should configure the Hadoop environment on all new data nodes (if you want to directly copy the hadoop folder that already exists on the node, you should delete the cluster data that has already been generated, such as the tmp and data directories, otherwise there will be problems)

Step 1: Turn off the firewall of the newly added data node.

Step 2: Complete the hosts information of the new node on each node

Step3: On the two NameNodes, open the tunnel to the new data node for keyless SSH login.

Step4: Append the hostnames of all new data nodes to the whitelist file specified by dfs.hosts on the two NameNodes. You need to change hdfs-site.xml. The blacklist file is used to exclude nodes in the cluster.

dfs.hosts /usr/hadoop/etc/hadoop/datanode-allow.list dfs.hosts.exclude /usr/hadoop/etc/hadoop/datanode-deny.list

Step5: Run the following refresh command on namenode: hdfs dfsadmin -refreshNodes

The current cluster node status can be viewed via hdfs dfsadmin -report or master:50070 web port

Step6: On the two NameNode nodes, change the slaves file and append the hostname of the data node to be launched to the slaves file. Start the datanode and node manager processes on slave:

sbin/hadoop-daemon.sh start datanode

sbin/yarn-daemon.sh start nodemanager

Use jps to view the process with new nodes. You can view the cluster situation through yarn node -list on master node.

Step 7: Balanced block(optional, set according to cluster situation)

./ bin/start-balancer.sh

1)If it is not balanced, the cluster will store all new data on the new node, which will reduce the efficiency of mapred.

2)Set balance threshold, default is 10%, the lower the value, the more balanced each node, but the longer the consumption time

hdfs balancer -threshold 5

3)Set the bandwidth of balance, default is only 1M/s

hdfs dfsadmin -setBalancerBandwidth 67108864

Data node deletion is similar to the above steps, except that datanode-allow.list and datanode-deni.list are modified on master1, as well as slaves files and synchronized to master2. The whole operation process does not involve the restart of the cluster ~ After waiting for the refresh operation, the machine that needs to be removed from the shelf can be safely shut down.

[root@master hadoop]# ./ bin/ hdfs dfsadmin -refreshNodes[root@master hadoop]# ./ bin/ hadoop dfsadmin -report

You can view the nodes connected to the current cluster

Decommission in progress: Decommission Status : Decommission in progress: Decommission Status : Decommissioned

1. Delete the corresponding machine in master's datanode-deny.list

2. Refresh node configuration on master:

hadoop dfsadmin -refreshNodes

3. Restart the tasktracker process on slave:

hadoop-daemon.sh start datanode

PS: If you haven't shut down the datastore process on the slave before, you need to shut it down and restart it.

The above is "How to dynamically add and delete data nodes in Hadoop 2.6 cluster" all the contents of this article, thank you for reading! I believe that everyone has a certain understanding, hope to share the content to help everyone, if you still want to learn more knowledge, welcome to pay attention to the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.