Shulou(Shulou.com)05/31 Report--

This article will explain in detail how to dynamically add nodes in Hadoop 2.6.0. The editor thinks it is very practical, so I share it for you as a reference. I hope you can get something after reading this article.

Basic preparation

In the basic preparation part, the main purpose is to set up the system environment in which hadoop is running.

Modify the system hostname (through hostname and / etc/sysconfig/network)

Modify the hosts file to configure the hosts of all nodes in the cluster (keep the hosts file uniform for all nodes in the cluster)

Set the password-free login of NameNode (required by both HA) to DataNode (implemented by ssh-copy-id command, which can be exempted from the permission modification after cp * .pub file)

Modify the slave file of the master node and add the ip information of the new node (used when the cluster is restarted)

Scp the configuration file of hadoop to the new node

Add DataNode

For the newly added DataNode node, you need to start the datanode process to add it to the cluster

On the new node, just run sbin/hadoop-daemon.sh start datanode

Then check the cluster situation through hdfs dfsadmin-report in namenode

Finally, you also need to set hdfs load balancing, because the default data transmission bandwidth is relatively low, which can be set to 64m, that is, hdfs dfsadmin-setBalancerBandwidth 67108864.

The default threshold of balancer is 10%, that is, the difference in total storage utilization between each node and the cluster is no more than 10%. We can set it to 5%.

Then start Balancer,sbin/start-balancer.sh-threshold 5 and wait for the cluster self-balancing to complete.

Add Nodemanager

Because Hadoop 2.x introduces the YARN framework, each compute node can be managed through NodeManager. Similarly, after starting the NodeManager process, it can be added to the cluster.

In the new node, you can run sbin/yarn-daemon.sh start nodemanager

In ResourceManager, check the cluster status through yarn node-list

This is the end of this article on "how to dynamically add nodes in Hadoop 2.6.0". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.