Shulou(Shulou.com)05/31 Report--

This article will explain in detail how to build hadoop pseudo-distribution for you. The editor thinks it is very practical, so I share it with you as a reference. I hope you can get something after reading this article.

First of all, you need to configure JDK, you can find many tutorials online.

Personal feeling is that when configuring the environment under Linux, the most important thing is to master some commands of shell and their format, otherwise you may encounter various command format errors and waste a lot of time.

Download jdk 1.7and store it in the "/ home/sr/setup/java-jdk-1.8/" directory. The downloaded file is "jdk-8u65-linux-x64.tar.gz", the version of java jdk is often updated, and the minor version number may be a little higher.

2 "sudo su -" ps: I use "sudo-I" here.

Switch to the root user and refer to "man sudo". This command switches to the root user, which is the most privileged user. Because what's going to be carried out later

The jdk installation is done in the / usr/local directory, and it is more convenient for users to use root.

3 "cd / usr/local/lib"

4 "tar-zxvf / home/sr/setup/java-jdk-1.8/jdk-8u65-linux-x64.tar.gz" ps: sr is the author's user name, please change it to your own!

Tar is a command-line tool for packaging and decompressing under linux. For details, please refer to "man tar". This command unzips the jdk-8u65-linux-x64.tar.gz package to the current directory. After unzipping, execute "ls", and you can see that there is a new directory under the current directory called "jdk 1.8.0. 65".

5 configure environment variable: 4.5.1 "gedit / etc/profile"

Gedit is the editor of notepad similar to Windoes under linux, and the file / etc/profile is the configuration file under linux. This command will open this.

Configuration file for editing.

6 add configuration

Add the following three lines of code at the end of the / etc/profile file: ps: be careful to change your JDK version

Export JAVA_HOME=/usr/local/lib/jdk1.8.0_65

Export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

Export PATH=$PATH:$JAVA_HOME/bin

7 "source / etc/profile" to make the configuration effective immediately

8 "java-version"

This command checks whether the jdk installation is successful.

Second, download hadoop2.7.1 and put it in the "/ home/sr/setup/hadoop" directory

Download link for hadoop stable version:

Http://mirror.esocc.com/apache/hadoop/common/stable/

1 "su sr"

Su command to switch users. Jdk is installed with the root user. Now switch back to the sr user.

2 "mkdir-p ~ / usr/hadoop" create the installation directory of Hadoop

3 "cd ~ / usr/hadoop"

4 after "tar-xvzf ~ / setup/hadoop/hadoop-2.7.1.tar.gz" is decompressed, there will be a directory ~ / usr/hadoop/hadoop-2.7.1, which is the home directory of hadoop

5 configure pseudo-distributed hadoop:

The configuration file is in the ~ / usr/hadoop/hadoop-2.7.1/etc/hadoop directory

Modify the contents of the core-site.xml file as follows: ps: note to add in the middle

Fs.default.name

Hdfs://localhost:9000

Modify the contents of the mapred-site.xml file as follows:

Mapred.job.tracker

Localhost:9001

Modify the contents of the hdfs-site.xml file as follows:

Dfs.replication

one

Add the following statement to the hadoop-env.sh file:

Export JAVA_HOME=/usr/local/lib/jdk 1.8.0_65

6 the command "sudo apt-get install ssh rsync" installs ssh and rsync. Ssh is a well-known secure shell protocol Secure Shell Protocol. Rsync is a command line tool for file synchronization.

7 "ssh-keygen-t dsa-f ~ / .ssh/id_dsa" execute this command to generate the public / private key of ssh. During execution, you will be prompted to enter characters and enter directly all the way.

8 "cat ~ / .ssh/id_dsa.pub > > ~ / .ssh/authorized_keys" ssh needs to enter a password when logging in remotely, but not if you use a public / private key. The way described above is to set up a public key / private key login.

9 when "ssh localhost" executes this command for the first time, a prompt appears, enter "yes" and enter.

10 "cd ~ / usr/hadoop/hadoop-2.7.1"

11 ". / bin/hadoop namenode-format" formats NameNode.

12 ". / sbin/start-all.sh" starts all nodes, including NameNode, SecondaryNameNode, JobTracker, TaskTracker, DataNode.

13 "jps" checks whether each process is running. At this point, you should see that there are six java virtual machine processes, namely Jps, NameNode, SecondaryNameNode,DataNode, JobTracker, TaskTracker, and see that six are correct, indicating that the startup is successful. If prompted that "jps" is not installed or cannot be found, execute "source / etc/profile" once.

Finally, let's run wordcount:

1 "cd ~ / usr/hadoop/hadoop-2.7.1"

2 create an input folder under "bin/hadoop fs-mkdir / input" HDFS

3 "bin/hadoop fs-put LICENSE.txt / input" put the LICENSE.txt in the folder

4 "bin/hadoop fs-ls / input" displays the contents of input

5 "bin/hadoop fs-cat / input/LICENSE.txt" to view the contents of the file

6 "bin/hadoop jar / home/sr/usr/hadoop/hadoop-2.7.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar wordcount / input / output" runs and outputs PS: please pay attention to the absolute path!

This is the end of the article on "how to build hadoop pseudo-distribution". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.