Shulou(Shulou.com)05/31 Report--

This article will give you a detailed explanation on how to tune Hadoop. The editor thinks it is very practical, so I share it with you as a reference. I hope you can get something after reading this article.

A HDFS-related tuning

The first parameter tuning for HDFS is the block size dfs.block.size. The default size of 64m Hadoop 2.x seems to have been changed to 128m. If the network environment is good, it is recommended to adjust it larger according to the actual business requirements.

For others, such as setting the number of backups for files, the default is 3, which is generally not recommended. One copy is saved on the datanode node, one on other nodes in the same rack, and the other on the other rack. By default, the data is randomly distributed on each node, but by setting rack awareness, the network transmission of data between nodes is reduced.

The configuration is as follows: configure hdfs-site.xml on the node of namenode

Topology.script.file.name / path/to/RackAware.py

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.