Shulou(Shulou.com)05/31 Report--

Editor to share with you how hadoop2.7.1 does not restart how to dynamically delete nodes and new nodes, I believe most people do not know much about it, so share this article for your reference, I hope you will learn a lot after reading this article, let's learn about it!

Restarting hadoop is too troublesome and needs to pause all current tasks. So it is important to delete and add nodes dynamically.

Let's start with dynamically deleting nodes:

(configured to delete nodes dynamically, this is to be restarted, and later, when nodes need to be deleted dynamically, there is no need to restart. ) enter the etc/hadoop/ directory of the hadoop directory

1. Add the excludes file and write in each line the hostname of the node to be deleted

two。 Add in hdfs-site.xml

Dfs.hosts.exclude/data/hadoop/etc/hadoop/excludes

3. Add in mapred-site.xml

Mapred.hosts.exclude/data/hadoop/etc/hadoop/excludestrue

In this way, if you need to delete a node, simply write the hostname of the node to the excludes file and execute it (just modify the file on namenode and execute the command)

Hadoop dfsadmin-refreshNodes

Looking at the node status hadoop dfsadmin-report or web management page, you can see that the node is in decomissioning status. When it changes to decomissioned, the node can be removed. After it is removed, all the processes on this node are still there. Just kill it.

Dynamically add nodes:

Because I am adding the deleted nodes back, there is no need to configure the environment and so on. Just use the original one.

Executed on the node, / data/hadoop/etc/hadoop/ is the configuration file directory, which is the startup datanode

Hadoop-daemon.sh-config / data/hadoop/etc/hadoop/-script hdfs start nodemanager

Then start nodemanager

Yarn-daemons.sh-config / data/hadoop/etc/hadoop/ start nodemanager

This command starts the nodemanager of all slave hosts, so an error will be reported during execution, and other hosts have been started.

I see that some posts are started by the following command:

Hadoop-daemon.sh start datanodeyarn-daemon.sh start nodemanager

When mine is activated, these two commands will not be tried. Friends can tell me after the test ~ (has been tested, successful, the second start is more reasonable)

Then, delete the hostname of the node in the previous excludes file and perform a refresh operation.

Hadoop dfsadmin-refreshNodes

At this point, you can see that the datanode of the node has been added.

Load balancing of hadoop data:

Execute the following command, and the parameter-threshold 5 can be left unwritten. The default is 10.

Start-banlancer.sh-threshold 5

Then you can check the log, and the log is on the machine on which the command is executed.

Tail-f / data/hadoop/logs/hadoop-hadoop-balancer-hostname.log

Wait until the equilibrium is over!

The above is all the contents of the article "how to dynamically delete nodes and add nodes without restarting hadoop2.7.1". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.