Shulou(Shulou.com)06/01 Report--

This article mainly explains the "Hadoop hot add and delete node steps", the content of the article is simple and clear, easy to learn and understand, the following please follow the editor's train of thought slowly in depth, together to study and learn "Hadoop hot add and delete node steps"!

Add nod

1. Modify host

Just like a normal datanode. Add ip for namenode

two。 Modify the configuration file conf/slaves of namenode

Add an ip or host for a new node

3. On the machine of the new node, start the service

[root@slave-004 hadoop] #. / bin/hadoop-daemon.sh start datanode [root @ slave-004 hadoop] #. / bin/hadoop-daemon.sh start tasktracker (hadoop1) [root@slave-004 hadoop] #. / bin/yarn-daemon.sh start nodemanager (hadoop2)

4. Balanced block

[root@slave-004 hadoop] #. / bin/start-balancer.sh

1) without balance, cluster will store all the new data on the new node, which will reduce the efficiency of mapred.

2) set the balance threshold. The default is 10%. The lower the value, the more balanced each node, but the longer the time consuming.

[root@slave-004 hadoop] #. / bin/start-balancer.sh-threshold 5

3) set the bandwidth (hdfs-site) of balance. Only 1M/s is available by default.

Dfs.balance.bandwidthPerSec 1048576 Specifies the maximum amount of bandwidth that each datanode can utilize for the balancing purpose in term of the number of bytes per second.

Note:

1. You must ensure that the firewall for slave is turned off

two。 Make sure that the ip of the new slave has been added to the / etc/hosts of master and other slaves, and vice versa, add the ip of master and other slave to the / etc/hosts of the new slave

3. If the start-balancer.sh still doesn't respond after running, make the-threshold as small as possible. The smaller the threshold, the less time it takes to balance it.

Start-balancer.sh-threshold 1

Hbase

If there is still hbase running on it, you need to deploy hbase's hserver.

5. Modify regionservers on hbasemaster

Vim / home/hadoop/hbase/conf/regionservers

Add a new node

Vim / home/hadoop/hbase/conf/hbase-site.xml

Add a new node to the hbase.zookeeper.quorum attribute

6. Copy the above two files to each node

7. Start hbase regionserver on the new node

Hbase-daemon.sh start regionserver

8. Start hbase shell in hbasemaster

Confirm the cluster situation with the status command

Delete nod

1. Cluster configuration

Modify the conf/hdfs-site.xml file

Dfs.hosts.exclude / data/soft/hadoop/conf/excludes Names a file that contains a list of hosts that are not permitted to connect to the namenode. The full pathname of the file must be specified. If the value is empty, no hosts are excluded.

Edit the conf/mapred-site.xml file and add the following configuration:

Mapred.hosts.exclude

/ opt/hadoop_conf/exclude_node

2 determine the machine to be removed from the shelf

The content of the file defined by dfs.hosts.exclude is one per line for each machine that needs to be offline. This will prevent them from connecting to Namenode. Such as:

Slave-003 slave-004

3. Force configuration reload

[root@master hadoop] #. / bin/hadoop dfsadmin-refreshNodes

It moves Block blocks in the background.

4. Close the node

After waiting for the operation to be finished, the machine that needs to be removed from the shelf can be safely shut down.

[root@master hadoop] #. / bin/ hadoop dfsadmin-report

You can view the nodes connected on the cluster now.

When Decommission is being executed, it will show: Decommission Status: Decommission in progress after execution, it will display: Decommission Status: Decommissioned

5. Edit the excludes file again

Once the machine is off the shelf, they can be removed from the excludes file

Log in to the machine to be taken off the shelf and you will find that the DataNode process is gone, but the TaskTracker still exists and needs to be handled manually

Thank you for reading, the above is the content of "Hadoop hot add and delete node steps". After the study of this article, I believe you have a deeper understanding of the steps of Hadoop hot add and delete nodes, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.