Shulou(Shulou.com)06/02 Report--

This article introduces the knowledge of "Hadoop rack perception and start-stop method". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Rack Awareness of Hadoop

The components of HDFS and Map/Reduce are rack aware.

NameNode and JobTracker obtain the rack id of each slave in the cluster by calling the APIresolve in the administrator configuration module. This API translates the DNS name (or IP address) of the slave into a rack id. Which module is used by configuring items

Specified by topology.node.switch.mapping.impl. The default implementation of the module invokes a script / command specified by the topology.script.file.name configuration item. If topology.script.file.name is not set, for all incoming IP addresses

The module returns / default-rack as the rack id There is also an additional configuration item mapred.cache.task.levels in the Map/Reduce section, which determines the number of stages of the cache (in the network topology). For example, if the default value is 2, two levels of cache- one for the host (host-> task mapping) and one level for the rack (rack-> task mapping) are established.

Start Hadoop

Starting the Hadoop cluster requires starting the HDFS cluster and the Map/Reduce cluster.

Format a new distributed file system:

$bin/hadoopnamenode-format

On the assigned NameNode, run the following command to start HDFS:

$bin/start-dfs.sh

The bin/start-dfs.sh script starts the DataNode daemon on all listed slave with reference to the contents of the ${HADOOP_CONF_DIR} / slaves file on NameNode.

On the assigned JobTracker, run the following command to start Map/Reduce:

$bin/start-mapred.sh

The bin/start-mapred.sh script starts the TaskTracker daemon on all listed slave with reference to the contents of the ${HADOOP_CONF_DIR} / slaves file on JobTracker.

Stop Hadoop

On the assigned NameNode, execute the following command to stop HDFS:

$bin/stop-dfs.sh

The bin/stop-dfs.sh script stops the DataNode daemon on all listed slave with reference to the contents of the ${HADOOP_CONF_DIR} / slaves file on NameNode.

On the assigned JobTracker, run the following command to stop Map/Reduce:

$bin/stop-mapred.sh

The bin/stop-mapred.sh script stops the TaskTracker daemon on all listed slave with reference to the contents of the ${HADOOP_CONF_DIR} / slaves file on JobTracker.

This is the end of the content of "Hadoop rack awareness and start-stop method". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.