Shulou(Shulou.com)06/03 Report--

Hadoop maintenance actual combat setup hdfs replica number dfs.replication

There is a temporary application, using a temporarily provisioned machine, the machine disk is tens of times different.

When using hadoop to store data, the utilization rate of small disk machines is more than 80%, while that of large disk machines is just over 10%.

Obviously, the utilization of datanode space in the hadoop cluster is very uneven and needs to be re-balance. So I thought of hdfs's equalizer balancer.

Sbin/start-balancer.sh-threshold 10

However, when the balancer of hdfs is executed, it automatically exits, does not execute, and no Error or Warning is found.

However, you can see some information in the out file:

Balancer has calculated the amount of data to move, but it's not moving, just a hint:

No block has been moved for 5 iterations. Exiting...

It turns out that hadoop balancer will judge when transferring block:

*

The balancer automatically exits when any of the following five

* conditions is satisfied:

*

* The cluster is balanced

* No block can be moved

* No block has been moved for five consecutive (continuous) iterations

* An IOException occurs while communicating with the namenode

* Another balancer is running.

*

* the following five situations can lead to the failure of Balance operations

* 1. The whole cluster has reached a balanced state

* 2. It is found that there are no block blocks that can be moved.

* 3. No block block was moved in 5 consecutive iterations

* 4. IO exception occurs when datanode node communicates with namenode node

* 5. A Balance operation already exists

Obviously, this is the third problem. Log shows that five iterations have been performed but each move should be due to multiple copies. The iterative block already exists on the large disk machine, and there is already there, so the balancer mobile process exits.

The equalizer cannot be executed, so the only way is to reduce the number of copies.

To modify the number of copies of hdfs, the first thought is to modify the dfs.replication parameter in hdfs-site.xml, which defaults to 3, initially set to 2, and now simply change it to 1, and then restart the datanode command: sbin/hadoop-daemon.sh start datanode

But it does not seem to work, the amount of data has not changed, it should be a large number of data reduction ah. And why is that?

First of all, the parameter dfs.replication is a client parameter, that is, the node level parameter. It needs to be set on each datanode.

A file, when uploaded to hdfs, specifies as many copies as possible. Later, if you change the number of copies, it will have no effect on the files that have already been uploaded. You can specify the number of copies to create while uploading the file.

If you only have 3 datanode, but you specify 4 copies, it will not work, because only one copy can be stored on each datanode.

Ah, that's the case. Let's take a look at the current number of copies of HDFS and modify and execute them. Is there a corresponding hadoop shell command?

Hadoop Commands and FileSystem Shell

Http://hadoop.apache.org/docs/r2.5.2/hadoop-project-dist/hadoop-common/CommandsManual.html

Http://hadoop.apache.org/docs/r2.5.2/hadoop-project-dist/hadoop-common/FileSystemShell.html

Execute hadoop fsck / to view the backup number distribution of the entire directory, and the output is as follows:

Default replication factor 2

Average block replication 2.38

Modify hdfs file backup factor: hadoop dfs-setrep [- R] if there is-R will modify the nature of the subdirectory file.

Hadoop dfs-setrep-w 1-R / user/hadoop/dir1 is to set the backup factor of all files in the directory to 1.

This takes some time, and it feels like it will scan all the files in the specified directory.

After modifying the stored files, remember to modify the number of data copies you put in later. This depends on the client side. If there are settings on the client side, be sure to modify them.

If the above is still unbalanced after execution, you can execute hdfs's equalizer balancer at this time.

I feel that the above is just a solution when there is a problem, and I hope that there will not be such a problem. After all, a large amount of data will have a great impact on the operation.

We should make a good plan for the value of dfs.replication and try to avoid the need to reduce it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.