Shulou(Shulou.com)06/02 Report--

In the previous article we covered Hadoop programming with an inverted index example based on MR programs, here we look at how to dynamically add and delete DataNodes in Hadoop.

Assuming the cluster operating system is CentOS 6.7 x64

Hadoop version: 2.6.3

1. Dynamically increase DataNode

1. Prepare a new DataNode machine, configure SSH mutual trust, and directly copy authorized_keys and id_rsa in the.ssh directory of an existing DataNode.

Copy Hadoop running directory, hdfs directory and tmp directory to new DataNode

Start hadoop on the new DataNode

.. sbin/hadoop-daemon.sh start datanode .. sbin/yarn-daemon.sh start datanode

Refresh nodes on NameNode

.. bin/hdfs dfsadmin -refreshNodes .. sbin/start-balancer.sh

5. To facilitate the next startup, you can add the domain name and ip of the new DataNode to/etc/hosts.

Second, dynamically delete DataNode

1. Configure hdfs-site.xml of NameNode, appropriately reduce the number of copies of dfs.replication, and increase dfs.hosts.exclude configuration.

dfs.hosts.exclude /usr/local/hadoop2/etc/hadoop/excludes

2. Create an excludes file under the corresponding path (/etc/hadoop/), and write the ip or domain name of the DataNode to be deleted.

Refresh all DataNodes on NameNode

.. bin/hdfs dfsadmin -refreshNodes .. sbin/start-balancer.sh

At this time, you can observe that DataNode gradually changes to Dead on the web detection interface (ip:50070).

summary

The above is the full content of this article on the dynamic addition and deletion of nodes in hadoop, hoping to help you. Interested friends can continue to refer to this site: talk about seven common Hadoop and Spark project cases, hadoop reformatting HDFS step analysis, etc., what questions can be directly left, Xiaobian will reply to everyone in time. Thank you for your support!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.