Shulou(Shulou.com)06/01 Report--

This article mainly introduces "the environment construction of hadoop stand-alone mode and pseudo-distribution mode under ubuntu12.04". In the daily operation, I believe that many people have doubts about the environment construction of hadoop stand-alone mode and pseudo-distribution mode under ubuntu12.04. I have consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the doubts about "the environment construction of hadoop stand-alone mode and pseudo-distribution mode under ubuntu12.04". Next, please follow the editor to study!

1. Add a hadoop group (specifically for hadoop testing) to the system user:

Sudo addgroup hadoop

two。 To add a hadoop user to hadoop:

Sudo adduser-ingroup hadoop hadoop

3. Give hadoop administrator privileges

Sudo usermod-aG sudo hadoop

(if you do not want to create new users, only build under your existing users, the above three steps can be omitted to directly enter the fourth step) 4 switch users! 5. Install the ssh remote login protocol:

Sudo apt-get install openssh-server

6. Start ssh

Sudo / etc/init.d/ssh start

7. Log in without a password and generate public and private keys:

Ssh-keygen-t rsa-P ""

The first operation will prompt for the password, press Enter directly, then two files will be generated under ~ / home/hadoop/.ssh: id_rsa and id_rsa.pub, the former is the private key and the latter is the public key. 8. Now let's append the public key to authorized_keys (authorized_keys is used to save all public key contents that are allowed to log in to the ssh client user as the current user)

Cat ~ / .ssh/id_rsa.pub > > ~ / .ssh/authorized_keys

9 Log in to ssh

Ssh localhost

10 exit

Exit

Log in and exit again, and then you don't have to enter a password to log in. 12 install java 13 install hadoop,hadoop official website download here choose hadoop-1.2.1.tar.gz, extract it and put it where you want to put it

Sudo mv / Downloads/hadoop-1.2.1.tar.gz / usr/local/hadoop

14 ensure that all operations are under hadoop users

Sudo chown-R hadoop:hadoop / usr/local/hadoop

15 configure hadoop-env.sh in the directory: / usr/local/hadoop/conf, add the following code

Export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-i386 (depending on your machine's java installation path) export HADOOP_HOME=/usr/local/hadoopexport PATH=$PATH:/usr/local/hadoop/binexport HADOOP_HOME_WARN_SUPPRESS= "TRUE"

16 let the configuration of environment variables take effect

Source / usr/local/hadoop/conf/hadoop-env.sh

17 the stand-alone configuration is successful:

Hadoop version

Pseudo-distribution mode continues: 18 configure three files under the conf directory: core-site.xml,hdfs-site.xml,mapred-site.xml core-site.xml:

Fs.default.name hdfs://localhost:9000 hadoop.tmp.dir / usr/local/hadoop/tmp

Hdfs-site.xml:

Dfs.replication 1 dfs.name.dir / usr/local/hadoop/hdfs/name dfs.data.dir / usr/local/hadoop/hdfs/data

Mapred-site.xml:

Mapred.job.tracker localhost:9001

19 create several new directories under the hadoop directory:

~ / hadoop$ mkdir tmp ~ / hadoop$ mkdir hdfs ~ / hadoop$ mkdir hdfs/name ~ / hadoop$ mkdir hdfs/data

20 modify the permissions of the data folder:

Sudo chmod GMurw / usr/local/hadoop/hdfs/data

21 make the configuration file effective:

Source / usr/local/hadoop/conf/hadoop-env.sh

twenty-two。 Verify that hadoop is not running:

Usr/local/hadoop bin/stop-all.sh

23 formatting namenode

Usr/local/hadoop bin/hadoop namenode-format

24 run

/ usr/local/hadoop bin/start-all.sh

25 jps if the following figure indicates success

26 Test program: run the example WordCount that comes with hadoop in pseudo-distribution mode to experience the following MapReduce process:

Note that the program runs on the file system dfs, and the files created are based on the file system:

First create the input directory in dfs

Hadoop@lcj:/usr/local/hadoop$ bin/hadoop dfs-mkdir input

Copy files from conf to input in dfs

Hadoop@lcj:/usr/local/hadoop$ bin/hadoop dfs-copyFromLocal conf/* input

Running WordCount in pseudo-distributed mode

Hadoop@lcj:/usr/local/hadoop$ bin/hadoop jar hadoop-examples-1.2.1.jar wordcount input output

Show output result

Hadoop@lcj:/usr/local/hadoop$ bin/hadoop dfs-cat output/*

When the Hadoop ends, you can use the stop-all.sh script to shut down the Hadoop daemon

Hadoop@lcj:/usr/local/hadoop$ bin/stop-all.sh at this point, on the "ubuntu12.04 hadoop stand-alone mode and pseudo-distribution mode of the environment to build" the end of the study, I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.