Shulou(Shulou.com)06/01 Report--

This article mainly introduces "the start and exit method of HDFS equilibrium operation". In the daily operation, I believe that many people have doubts about the start and exit method of HDFS equilibrium operation. The editor consulted all kinds of information and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts about "start and exit method of HDFS equilibrium operation". Next, please follow the editor to study!

Start of HDFS equalization operation

Using HDFS's balancer command, you can configure a Threshold to balance each DataNode disk utilization. The command is as follows:

Start-balancer.sh-threshold 8

After running, a Balancer process appears:

The above command sets Threshold to 8%. When executing the balancer command, first count the average disk utilization of all DataNode, and then judge that if the disk utilization of a DataNode exceeds this average Threshold, then the block of this DataNode will be transferred to DataNode with low disk utilization, which is very useful for the addition of new nodes. The value of Threshold is between 1 and 100. if the parameter is not displayed, the default is 10.

When the scope is out of range, an exception will be thrown:

Java.lang.IllegalArgumentException: Numberout of range: threshold = 0.07

Atorg.apache.hadoop.hdfs.server.balancer.Balancer$Cli.parse (Balancer.java:1535)

Atorg.apache.hadoop.hdfs.server.balancer.Balancer$Cli.run (Balancer.java:1510)

Atorg.apache.hadoop.util.ToolRunner.run (ToolRunner.java:70)

Atorg.apache.hadoop.hdfs.server.balancer.Balancer.main (Balancer.java:1582)

2012-12-19 16 2833 299 ERRORorg.apache.hadoop.hdfs.server.balancer.Balancer: Exiting balancer due anexception

Java.lang.IllegalArgumentException: Numberout of range: threshold = 110.0

Atorg.apache.hadoop.hdfs.server.balancer.Balancer$Cli.parse (Balancer.java:1535)

Atorg.apache.hadoop.hdfs.server.balancer.Balancer$Cli.run (Balancer.java:1510)

Atorg.apache.hadoop.util.ToolRunner.run (ToolRunner.java:70)

Atorg.apache.hadoop.hdfs.server.balancer.Balancer.main (Balancer.java:1582)

The smaller the parameter value, the longer it takes. When using this command, the block will be transferred repeatedly from the node with high disk utilization to the disk with low disk utilization, no more than 10G at a time and no more than 20 minutes at a time.

When making equalization, it will affect the network bandwidth. You can limit the bandwidth of equalization operation in the configuration file:

Dfs.balance.bandwidthPerSec

1048576

Specifies themaximum bandwidth that each datanode can utilize for the balancing purpose interm of the number of bytes per second.

If it is not set, the speed defaults to the 1M/S size during balance operation. Takes effect when the parameter is restarted. Multiple equalizers are not allowed to operate at the same time in a cluster.

Exit of HDFS equalization operation

In addition to using the stop-balancer.sh script directly on the command line to perform the exit equalization operation, the currently performed equalization operation will also exit when the following occurs:

(1) the cluster has reached equilibrium.

(2) No block can be moved again.

(3) there is no block movement in five consecutive iterations.

(4) IOException appears when communicating with NameNode

(5) another equalization operation starts.

At this point, the study on "the start and exit method of HDFS equilibrium operation" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.