Shulou(Shulou.com)06/01 Report--

This article mainly shows you "how to install Hadoop cluster", the content is easy to understand, clear, hope to help you solve your doubts, let the editor lead you to study and learn "how to install Hadoop cluster" this article.

I. installation environment

Because the number of hosts needed to install the Hadoop cluster must be odd, here I built three hosts with virtual machines. To prevent IP changes, IP should be set to static IP.

IP

Hostnam

Operating system

Action

192.168.121.201

Master.Hadoop

CentOS6.6_x64

NameNode

ResourceManager

SecondaryNameNode

192.168.121.202

Slave1.Hadoop

CentOS6.6_x64

DataNode

NodeManager

192.168.121.203

Slave2.Hadoop

CentOS6.6_x64

DataNode

NodeManager

II. Required software

1 JDK

You can download JDK or use the OpenJDK that comes with the CentOS system.

If you use the OpenJDK that comes with the system, just set and activate it in / etc/profile:

# vi / etc/profile

Export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64

Export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

Export PATH=$PATH:$JAVA_HOME/bin

Make the variable take effect immediately and test it

# source / etc/profile

# echo $JAVA_HOME

/ usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64

2 Hadoop

Download address: http://hadoop.apache.org/releases.html

What I downloaded here is the 2.8.0 binary version.

Configure password-free login between hosts

Add to the / etc/hosts of each host

192.168.121.201 Master.Hadoop

192.168.121.202 Slave1.Hadoop

192.168.121.203 Slave2.Hadoop

Next, refer to http://blog.chinaunix.net/uid-29485627-id-4232161.html

IV. Hadoop cluster installation

1 extract hadoop and move to / usr/Hadoop

[root@Master Desktop] # mv hadoop-2.8.0.tar.gz / usr/

[root@Master Desktop] # cd / usr/

[root@Master usr] # tar-zxvf hadoop-2.8.0.tar.gz

[root@Master usr] # mv hadoop-2.8.0 hadoop

2 create a folder where temporary files are stored in hadoop

# cd / usr/hadoop

# mkdir tmp

3 modify the configuration file

(1) configure JAVA_HOME in hadoop-env.sh

[root@Master usr] # cd etc/hadoop/

[root@Master hadoop] # vi hadoop-env.sh

Export JAVA_HOME=$ {JAVA_HOME}

Change to export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64

(2) configure core-site.xml

Fs.defaultFS

Hdfs://Master.Hadoop:9000

Io.file.buffer.size

131072

Hadoop.tmp.dir

/ usr/hadoop/tmp

(3) configure hdfs-site.xml

Dfs.namenode.secondary.http-address

Master.Hadoop:50090

Dfs.replication

two

Dfs.namenode.name.dir

File:/usr/hadoop/hdfs/name

Dfs.datanode.data.dir

File:/usr/hadoop/hdfs/data

(4) configure yarn-site.xml

Yarn.nodemanager.aux-services

Mapreduce_shuffle

Yarn.resourcemanager.address

Master.Hadoop:8032

Yarn.resourcemanager.scheduler.address

Master.Hadoop:8030

Yarn.resourcemanager.resource-tracker.address

Master.Hadoop:8031

Yarn.resourcemanager.admin.address

Master.Hadoop:8033

Yarn.resourcemanager.webapp.address

Master.Hadoop:8088

(5) configure mapred-site.xml

Mapreduce.framework.name

Yarn

Mapreduce.jobhistory.address

Master.Hadoop:10020

Mapreduce.jobhistory.address

Master.Hadoop:19888

(6) configure slaves file

[root@Master hadoop] # vi slaves

Slave1.Hadoop

Slave2.Hadoop

4 copy the package configured by hadoop to another host

# scp-r / usr/hadoop Slave1.Hadoop:/usr/

# scp-r / usr/hadoop Slave2.Hadoop:/usr/

5 configure Hadoop environment variables on three machines

# vi / etc/profile

Export HADOOP_HOME=/usr/hadoop

Export PATH=$PATH:$HADOOP_HOME/bin

Make the variable take effect immediately and verify:

# source / etc/profile

# echo $HADOOP_HOME

# echo $PATH

Fifth, format the node

Execute on the Master.Hadoop host

# cd / usr/hadoop/bin

#. / hdfs namenode-format

VI. All Hadoop clusters are started

1 execute on the Master.Hadoop host

# cd / usr/hadoop/sbin

#. / start-dfs.sh

#. / start-yarn.sh

2 execute jps on the Master.Hadoop host

[root@Master sbin] # jps

6348 NameNode

6694 ResourceManager

6960 Jps

6546 SecondaryNameNode

3 execute jps on the Slave1.Hadoop host

[root@Slave1 hadoop] # jps

6016 DataNode

6251 Jps

6126 NodeManager

4 execute jps on the Slave2.Hadoop host

[root@Slave2 hadoop] # jps

5815 DataNode

6051 Jps

5923 NodeManager

5 casually open a firefox browser on three CentOS hosts and enter http://192.168.121.201:8088

These are all the contents of the article "how to install Hadoop clusters". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.