Shulou(Shulou.com)06/01 Report--

This article mainly shows you "centos6.5 how to install hadoop1.2.1", the content is easy to understand, clear, hope to help you solve your doubts, the following let the editor lead you to study and learn "centos6.5 how to install hadoop1.2.1" this article.

This article only briefly introduces the installation steps

1. Role assignment

10.11.84.4 web-crawler--1.novalocal master/slave10.11.84.5 web-crawler--2.novalocal slave10.11.84.6 web-crawler--3.novalocal slave

two。 Installation directory / usr/local/hadoop (HADOOP_HOME)

3, create group hadoop

Groupadd hadoop

4. Create user hadoop

Useradd-d / home/hadoop-g hadoop hadoopchown-R hadoop:hadoop / usr/local/hadoop

5. Install jdk on each machine (installed in the same directory)

1) if you check the java version, you can see that jdk is openjdk, and it is best to replace it with sun jdk.

[hadoop@hadoop2 ~] $java-versionopenjdk version "1.8.0mm 131" OpenJDK Runtime Environment (build 1.8.0_131-b12) OpenJDK 64-Bit Server VM (build 25.131-b12) Mixed mode) [hadoop@hadoop2 ~] $rpm-qa | grep javajava-1.8.0-openjdk-headless-1.8.0.131-11.b12.el7.x86_64javapackages-tools-3.4.1-11.el7.noarchtzdata-java-2017b-1.el7.noarchjava-1.7.0-openjdk-headless-1.7.0.141-2.6.10.5.el7.x86_64java-1.7.0-openjdk-1.7.0.141-2.6 .10.5.el7.x86 _ 64java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64python-javapackages-3.4.1-11.el7.noarch

2) Uninstall openjdk

Rpm-qa | grep java | xargs yum-y remove

3) install sunjdk

Extract jdk-8u181-linux-i586.tar.gz under / usr/local directory and name it jdk

Configure the environment variables as follows

Add the following at the end of vim / etc/profile

Export JAVA_HOME=/usr/local/jdkexport JRE_HOME=$ {JAVA_HOME} / jreexport PATH=$ {JAVA_HOME} / bin:$ {PATH} export CLASSPATH=.:$ {JAVA_HOME} / lib:$ {JRE_HOME} / lib:$ {CLASSPATH}

4) check whether jdk is installed successfully

[root@hadoop2 local] # java-versionjava version "1.8.0mm 151" Java (TM) SE Runtime Environment (build 1.8.0_151-b12) Java HotSpot (TM) 64-Bit Server VM (build 25.151-b12, mixed mode)

6. Login from master to slave ssh without password

Execute the following command under the hadoop user on each machine

[hadoop@hadoop1 ~] $ssh-keygen-t rsa-P''# directly enter and send master machine ~ / .ssh/id_rsa.pub to all slave nodes Scp id_rsa.pub hadoop@10.11.84.5:~Scp id_rsa.pub hadoop@10.11.84.6:~ and execute the following command under each slave node hadoop user: Cd ~ / .sshCat. / id_rsa.pub > > authorized_keysChmod 644 authorized_keys

7. Turn off all machine firewalls, selinux

1) below centos 7

Firewalls

View firewall status

Service iptables status

Temporarily turn off the firewall

Service iptables stop

Permanently turn off the firewall

Chkconfig iptables off

Selinux

Setenforce 0 # temporarily shut down

Permanent closure requires the following files to be modified

Vim / etc/selinux/config

Change SELINUX=enforcing to SELINUX=disabled

2) centos 7

Firewall-cmd-- reload # restart firewall

Systemctl stop firewalld.service # stop firewall

Systemctl disable firewalld.service # prevents firewall from booting

Firewall-cmd-- state # View the default firewall status (show notrunning when turned off and running when turned on)

Selinux ditto

8. Modify hadoop configuration file

Core-site.xmlhadoop.tmp.dir/home/hadoop/tmpAbase for other temporary directories.fs.default.namehdfs://web-crawler--1.novalocal:9000hadoop-env.sh add java_home export JAVA_HOME=/usr/local/jdkhdfs-site.xmldfs.replication2dfs.name.dir/home/hadoop/namedfs.data.dir/home/hadoop/datamapred-site.xml mapred.job.tracker http://web-crawler--1.novalocal:9001;;; Mapred.system.dir / home/hadoop/mapred/system mapred.local.dir / home/hadoop/mapred/local

5) masters

Web-crawler--1.novalocal

6) slaves

Web-crawler--1.novalocalweb-crawler--2.novalocal web-crawler--3.novalocal

9. Modify environment variables to add HADOOP_HOME and JAVA_HOME

Vim / etc/profile (root) export ANT_HOME=/usr/local/antexport PATH=$ {PATH}: ${ANT_HOME} / binexport HADOOP_HOME=/usr/local/hadoop-1.2.1export JAVA_HOME=/usr/local/jdkexport PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATHexport CLASSPATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATHexport PATH=/usr/local/lib/phantomjs/bin:$PATHexport ANT_HOME=/usr/local/antexport PATH=$ {PATH}: ${ANT_HOME} / binexport PATH=$PATH:$ {HADOOP_HOME} / bin:$ {HADOOP_HOME} / sbin

10. Start hadoop

1) format namenode on master

Hadoop namenode-format

2) start hadoop

Sh / user/local/hadoop/bin/start-all.sh

11. Conduct a simple test

Hadoop fs-mkdir-p / user/hadoop/private/inHadoop fs-put / user/local/hadoop/bin/start-all.sh / user/hadoop/private/inCd / usr/local/hadoopHadoop jar hadoop-examples-1.2.1.jar wordcount / user/hadoop/private/in / user/hadoop/private/out are all the contents of the article "how to install hadoop1.2.1 in centos6.5". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.