Shulou(Shulou.com)06/02 Report--

This article will explain in detail how to install Hadoop under Linux. The editor thinks it is very practical, so I share it with you for reference. I hope you can get something after reading this article.

Install Hadoop tutorial under Linux

Purpose: this installation Hadoop tutorial is designed to guide beginners who are not exposed to Hadoop for entry-level purposes only, master please float by.

Environment: Fedora12

Kernel2.6.31.5-127.fc12.i686.PAE

Hadoop-0.20.2

Jdk-6u18-linux-i586

Beginners can choose three physical machines to do experiments, one for NameNode,JobTracker, the other two for DataNode,TaskTracker.

The three machines used in this installation Hadoop tutorial do the demonstration.

Station1192.168.128.11NameNode

Station2192.168.128.22DataNode

Station3192.168.128.33DataNode

1. There is no need to demonstrate the installation of Fedora. After installation, the sshd service is started by default.

If you are not sure, you can check it by hand.

[root@station1~] # servicesshdstatus

If it doesn't start, you can start it manually.

[root@station1~] # servicesshdstart

2. Establish ssh login without password

On NameNode

[cuijj@station1~] $ssh-keygen-tdsa-P''-f~/.ssh/id_dsa

Two files are generated in ~ / .ssh /: id_dsa and id_dsa.pub. These two appear in pairs.

Append the id_dsa.pub file to the authorized_keys on DataNode (there is no such file by default).

[cuijj@station1~] $scpid_dsa.pubcuijj@192.168.128.22:/home/cuijj/

Log in to 192.168.128.22 (same as another DataNode)

[cuijj@station1~] $catid_dsa.pub > > ~ / .ssh/authorized_keys

3. Turn off the firewall

[root@station1~] # serviceiptablesstop

4. Install jdk1.6 (make sure that several machines have the same java environment)

Download jdk-6u18-linux-i586.bin from the official website http://java.sun.com. After downloading, install it directly. The installation path for this installation Hadoop tutorial is / home/cuijj/jdk1.6.0_18. Add the following statement to / ect/profile after installation:

ExportJAVA_HOME=/home/cuijj/jdk1.6.0_18

ExportJRE_HOME=/home/cuijj/jdk1.6.0_18/jre

ExportCLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH

ExportPATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH

5. Install hadoop

Download hadoop-0.20.2.tar.gz from the official website

[cuijj@station1~] $tarxzvfhadoop-0.20.2.tar.gz

Add the installation path of Hadoop to / etc/profile

ExportHADOOP_HOME=/home/cuijj/hadoop-0.20.2

ExportPATH=$HADOOP_HOME/bin:$PATH

For / etc/profile to take effect, source

[cuijj@station1~] $source/etc/profile

6. Configure hadoop

The installation Hadoop tutorial describes how to configure Hadoop. The configuration file for hadoop is in the / conf directory

1) configure Java environment

[cuijj@station1~] $vimhadoop-0.20.2/conf/hadoop-env.sh

ExportJAVA_HOME=/home/cuijj/jdk1.6.0_18

2) configure conf/core-site.xml, conf/hdfs-site.xml, conf/mapred-site.xml files

[cuijj@station1~] $vimhadoop-0.20.2/conf/core-site.xml

[cuijj@station1~] $vimhadoop-0.20.2/conf/mapred-site.xml

[cuijj@station1~] $vimhadoop-0.20.2/conf/hdfs-site.xml

3) copy the complete hadoop from NameNode to DataNode

4) configure conf/masters and conf/slaves on NameNode

Masters:

192.168.128.11

Slaves:

192.168.128.22

192.168.128.33

7. Run hadoop

1) format the file system

[cuijj@station1hadoop-0.20.2] $hadoopnamenode-format

2) start Hadoop

[cuijj@station1hadoop-0.20.2] $bin/start-all.sh

3) use the jps command to view the process, and the result on NameNode is as follows:

4) View cluster status

[cuijj@station1hadoop-0.20.2] $hadoopdfsadmin-report

5) use hadoop's web method to view

[cuijj@station1hadoop-0.20.2] $links http://192.168.128.11:50070

8. Run wordcount.java program

1) create two files file01 and file02 on the local disk first

[cuijj@station1~] $echo "Hellocuijjbyecuijj" > file01

[cuijj@station1~] $echo "HelloHadoopGoodbyeHadoop" > file02

2) create an input directory in hdfs

[cuijj@station1~] $hadoopdfs-mkdirinput

3) copy file01 and file02 to the input directory of hdfs

[cuijj@station1~] $hadoopdfs-copyFromLocal/home/cuijj/file0*input

4) check whether there is an input directory in hdfs

[cuijj@station1~] $hadoopdfs-ls

5) check whether file01 and file02 have been successfully copied in the input directory

6) execute wordcount (make sure there is no output directory on hdfs)

[cuijj@station1hadoop-0.20.2] $hadoopjarhadoop-0.20.2-examples.jarwordcountinputoutput

7) complete the run and view the result

On "how to install Hadoop under Linux" this article is shared here, I hope the above content can be of some help to you, so that you can learn more knowledge, if you think the article is good, please share it out for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.