Shulou(Shulou.com)06/02 Report--

This article mainly explains "the process of building hadoop distributed cluster". The content of the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "the process of building hadoop distributed cluster".

Hadoop distributed cluster building: 1. Cluster planning:

two。 Specific steps: (1) upload the installation package and extract it to the relevant directory:

[hadoop@hadoop01 home] $put c:/hadoop-2.6.5-centos-6.7.tar.gz

[hadoop@hadoop01 home] $tar-zxvf hadoop-2.6.5-centos-6.7.tar.gz-C / home/hadoop/apps

(2) modify the configuration file:

[hadoop@hadoop01 home] $cd / home/hadoop/apps/hadoop-2.6.5/hadoop/etc

Here are six configuration files that need to be modified:

Hadoop-env.sh:

Join: export JAVA_HOME=/usr/java/jdk1.8.0_73

Core-site.xml:

Fs.defaultFS hdfs://hadoop01:9000 hadoop.tmp.dir / home/hadoop/hadoopdata

Hdfs-site.xml:

In order to ensure the security of metadata, dfs.namenode.name.dir / home/hadoop/hadoopdata/name generally configures the data storage of multiple different directories dfs.datanode.data.dir / home/hadoop/hadoopdata/datadatanode, the replica storage number of data blocks of dfs.replication 2HDFS, the information of dfs.secondary.http.address hadoop02:50090secondarynamenode running nodes, and different nodes of namenode

Mapred-site.xml:

Mapreduce.framework.name yarn

Yarn-site.xml:

Yarn.resourcemanager.hostname hadoop03 yarn.nodemanager.aux-services mapreduce_shuffleYARN cluster provides shuffle services for MapReduce programs

Slaves:

Hadoop01 hadoop02 hadoop03 (3) distributes the installation package to each node, and each node of the Hadoop cluster needs to install the Hadoop installation package:

[hadoop@hadoop01 etc] $scp-r hadoop-2.6.5 hadoop02:$PWD

[hadoop@hadoop01 etc] $scp-r hadoop-2.6.5 hadoop03:$PWD

(4) configure the environment variables of hadoop

[hadoop@hadoop01 etc] $sudo vim / etc/profile:

Join:

Export HADOOP_HOME=/home/hadoop/apps/hadoop-2.6.5/

Export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

[hadoop@hadoop01 etc] $source / etc/profile

(5) execute the command on the HDFS master node to initialize the namenode

[hadoop@hadoop01 etc] $hadoop namenode-format

(6) start HDFS on the primary node on HDFS

[hadoop@hadoop01 etc] $start-dfs.sh

(7) start YARN on the primary node on HDFS

[hadoop@hadoop01 etc] $sbin/start-yarn.sh

Add:

The web interface of hdfs is: http://hadoop01:50070

The web interface of yarn is: http://hadoop03:8088

Check the status of the cluster: hdfs dfsadmin-report

1. Start namenode or datenode

Sbin/hadoop-daemon.sh start datanode

Sbin/hadoop-daemon.sh start namenode

Sbin/hadoop-daemon.sh start secondarynamenode

2. Start yarn nodemanager

Sbin/yarn-daemon.sh start nodemanager

Sbin/yarn-daemon.sh start resourcemanager

Thank you for your reading. the above is the content of "the process of building hadoop distributed cluster". After the study of this article, I believe you have a deeper understanding of the process of building hadoop distributed cluster, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.