Shulou(Shulou.com)06/02 Report--

This article mainly introduces the example analysis of Hadoop setting in the process of implementing Hadoop by Cloudera. It is very detailed and has a certain reference value. Interested friends must read it!

Set the Hadoop of the primary node

Set up JAVA_HOME

The JAVA_HOME of Hadoop is set in the file / etc/conf/hadoop-env.sh, as shown below:

Sudo vi / etc/conf/hadoop-env.shexport JAVA_HOME= "/ usr/lib/jvm/java-6-sun"

Core configuration of Hadoop

The core configuration file of Hadoop is / etc/hadoop/conf/core-site.xml, which is configured as follows:

Fs.default.namehdfs://hadoop-01:8020hadoop.tmp.dir/var/lib/hadoop-0.20/cache/$ {user.name}

Set up a distributed storage environment for Hadoop

During the implementation of Hadoop by Cloudera, the distributed environment setting of Hadoop is mainly realized through file / etc/hadoop/conf/hdfs-site.xml. The specific configuration is as follows:

Dfs.replication3dfs.permissionsfalsedfs.name.dir/var/lib/hadoop-0.20/cache/hadoop/dfs/name

Set up a distributed computing environment for Hapoop

In the process of implementing Hadoop by Cloudera, Map/Reduce algorithm is used in the distributed computing of Hadoop. The setting of the algorithm environment is mainly realized through file / etc/hadoop/conf/mapred-site.xml. The specific configuration is as follows:

Mapred.job.trackerhadoop-01:8021

Set the master-slave node of Hadoop

First set up the master node and edit the / etc/hadoop/conf/masters file, as shown below:

Hadoop-01 then sets the slave node and edits the / etc/hadoop/conf/slaves file as follows:

Hadoop-02hadoop-03hadoop-04firehare-303

Set the Hadoop on the slave node

The Hadoop setting on the slave node is very simple, you only need to copy a copy of the Hadoop setting on the master node to the slave node.

Scp-r / etc/hadoop/conf hadoop-02:/etc/hadoopscp-r / etc/hadoop/conf hadoop-03:/etc/hadoopscp-r / etc/hadoop/conf hadoop-04:/etc/hadoopscp-r / etc/hadoop/conf firehare-303:/etc/hadoop. Cloudera needs to start the Hadoop service after configuring Hadoop in the process of implementing Hadoop.

Start Hadoop

Format distributed file system

There is another preparatory work to be done before starting Hadoop, that is, formatting the distributed file system, which only needs to be done on the master node, as shown below:

/ usr/lib/hadoop-0.20/bin/hadoop namenode-format

Start the Hadoop service

Starting Hadoop can be achieved by the following command:

/ usr/lib/hadoop-0.20/bin/start-all.sh Note: this command does not add sudo, if you add sudo, it will prompt an error message, because root users do not have no authentication ssh setting. The following is the output information. Note that the hadoop-03 did not answer on purpose, so the No route to host message appears. In this way, Hadoop will start normally!

Test Hadoop

Once the Hadoop is set up, the next step is to test it to see if it works. If the test is successful, it means that the Hadoop is deployed successfully and can be used for Map/Reduce distribution calculation.

The above is all the contents of the article "sample Analysis of Hadoop Settings in the process of Cloudera implementing Hadoop". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.