Shulou(Shulou.com)06/03 Report--

First, download

1. Download hadoop

[root@master ~] # cd / usr/local/src/

[root@master src] # wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz

two。 Download address on the official website (latest available)

Http://hadoop.apache.org/releases.html#Download

two。 Installation

1. Decompression

[root@master src] # tar zxvf hadoop-2.7.2.tar.gz

two。 move

[root@master src] # mv hadoop-2.7.2 / usr/local/

3. Copy to all slave

[root@master src] # rsync-av / usr/local/hadoop slave1:/usr/local/

[root@master src] # rsync-av / usr/local/hadoop slave2:/usr/local/

[root@master src] # rsync-av / usr/local/hadoop slave3:/usr/local/

[root@master src] # rsync-av / usr/local/hadoop slave4:/usr/local/

[root@master src] # rsync-av / usr/local/hadoop slave5:/usr/local/

[root@master src] # rsync-av / usr/local/hadoop slave6:/usr/local/

III. Configuration

1. Create a directory

[root@master] # mkdir-p / data/hadoop

[root@master ~] # cd! $& & mkdir tmp dfs dfs/data dfs/name

Do the same on all slave.

two。 Configure the core-site.xml file

[root@master hadoop] # vim / usr/local/hadoop/etc/hadoop/core-site.xml

Add the following:

Fs.defaultFS

Hdfs://192.168.0.194:9000

Hadoop.tmp.dir

File:/data/hadoop/tmp/

Io.file.buffer.size

131702

Note:

The IP after ① hdfs is the ip of master.

The path followed by ② file is to create the tmp directory path

3. Configure the hdfs-site.xml file

[root@master hadoop] # vim / usr/local/hadoop/etc/hadoop/hdfs-site.xml

Add the following:

Dfs.namenode.name.dir

File:/data/hadoop/dfs/name

Dfs.datanode.data.dir

File:/data/hadoop/dfs/data

Dfs.replication

two

Dfs.namenode.secondary.http-address

192.168.0.194:9001

Dfs.webhdfs.enabled

True

4. Configure the mapred-site.xml file

[root@master hadoop] # cp / usr/local/hadoop/etc/hadoop/mapred-site.xmll.template / usr/local/hadoop/etc/hadoop/mapred-site.xml

[root@master hadoop] # vim / usr/local/hadoop/etc/hadoop/mapred-site.xml

Add the following:

Mapreduce.framework.name

Yarn

Mapreduce.jobhistory.address

192.168.0.194:10020

Mapreduce.jobhistory.webapp.address

192.168.0.194:19888

5. Configure the yarn-site.xml file

[root@master hadoop] # vim / usr/local/hadoop/etc/hadoop/yarn-site.xml

Add the following:

Yarn.nodemanager.aux-services

Mapreduce_shuffle

Yarn.nodemanager.auxservices.mapreduce.shuffle.class

Org.apache.hadoop.mapred.ShuffleHandler

Yarn.resourcemanager.address

192.168.0.194:8032

Yarn.resourcemanager.scheduler.address

192.168.0.194:8030

Yarn.resourcemanager.resource-tracker.address

192.168.0.194:8031

Yarn.resourcemanager.admin.address

192.168.0.194:8033

Yarn.resourcemanager.webapp.address

192.168.0.194:8088

Yarn.nodemanager.resource.memory-mb

2048

Note:

2048 means to set the memory size. More than 2 gigabytes is recommended, otherwise the interface can be jammed in the analysis.

6. Modify the configuration hadoop-env.sh file

[root@master hadoop] # vim / usr/local/hadoop/etc/hadoop/hadoop-env.sh

Modify the following:

Will: export JAVA_HOME=$ {JAVA_HOME}

Changed to: export JAVA_HOME=/usr/local/jdk1.7.0_79

7. Modify the configuration yarn-env.sh file

[root@master hadoop] # vim / usr/local/hadoop/etc/hadoop/yarn-env.sh

Add the following before the first if:

Export JAVA_HOME=/usr/local/jdk1.7.0_79

8. Modify the configuration slaves file

[root@master hadoop] # vim / usr/local/hadoop/etc/hadoop/slaves

Add all the IP of slave into, for example:

192.168.0.2

192.168.0.3

.

9. Synchronize profile

[root@master hadoop] # rsync-av / usr/local/hadoop/etc/ slave1:/usr/local/hadoop/etc

[root@master hadoop] # rsync-av / usr/local/hadoop/etc/ slave2:/usr/local/hadoop/etc

[root@master hadoop] # rsync-av / usr/local/hadoop/etc/ slave3:/usr/local/hadoop/etc

[root@master hadoop] # rsync-av / usr/local/hadoop/etc/ slave4:/usr/local/hadoop/etc

[root@master hadoop] # rsync-av / usr/local/hadoop/etc/ slave5:/usr/local/hadoop/etc

[root@master hadoop] # rsync-av / usr/local/hadoop/etc/ slave6:/usr/local/hadoop/etc

IV. Start the service

1. Initialization

[root@master hadoop] # / usr/local/hadoop/bin/hdfs namenode-format

[root@master hadoop] # echo $?

0

Note: 0 means there is no error. If the natural number is greater than 0, it indicates the initialization error. See the output to solve the problem.

two。 Start the service (yarn)

[root@master hadoop] # / usr/local/hadoop/sbin/start-yarn.sh

Starting yarn daemons

Starting resourcemanager, logging to / usr/local/hadoop/logs/yarn-root-resourcemanager-master.out

192.168.0.175: starting nodemanager, logging to / usr/local/hadoop/logs/yarn-root-nodemanager-slave1.out

192.168.0.191: starting nodemanager, logging to / usr/local/hadoop/logs/yarn-root-nodemanager-slave6.out

192.168.0.176: starting nodemanager, logging to / usr/local/hadoop/logs/yarn-root-nodemanager-slave2.out

192.168.0.184: starting nodemanager, logging to / usr/local/hadoop/logs/yarn-root-nodemanager-slave5.out

192.168.0.178: starting nodemanager, logging to / usr/local/hadoop/logs/yarn-root-nodemanager-slave3.out

192.168.0.183: starting nodemanager, logging to / usr/local/hadoop/logs/yarn-root-nodemanager-slave4.out

Note: version 2.7.1 starts the service with a script called start-all.sh (2.7.2 has been deprecated)

3. Stop service (yarn)

[root@master hadoop] # / usr/local/hadoop/sbin/stop-yarn.sh

4. Start the service (hdfs)

[root@master ~] # / usr/local/hadoop/sbin/start-dfs.sh

5. Stop service (hdfs)

[root@master ~] # / usr/local/hadoop/sbin/stop-dfs.sh

Check on 6.slave to see if hadoop is started

[root@slave1 ~] # ps aux | grep java

[root@slave1 ~] # netstat-nlp | grep java if there is no yum-y install net-tools

5. Browser access

Http://192.168.0.194:8088

Http://192.168.0.194:50070

5. Testing

1. Create a directory

[root@master ~] # cd / usr/local/hadoop/

[root@master hadoop] # bin/hdfs dfs-mkdir / 123

two。 View the catalog

[root@master hadoop] # bin/hdfs dfs-ls /

3. Copy test files

[root@master hadoop] # bin/hdfs dfs-copyFromLocal. / LICENSE.txt / 123

4. Analyze files (statistics files with more words and fewer words)

[root@master hadoop] # bin/hadoop jar. / share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount / 123/LICENSE.txt / 111/123

/ 111cus123: indicates the directory where the analysis results are stored

Jar: indicates what type of file to use

2.7.2.jar: indicates the specific file used, this is the best TAB completion, different versions have different names.

5. View analysis results

[root@master hadoop] # bin/hdfs dfs-cat / 111/123/part-r-00000

6. Delete directory

[root@master hadoop] # bin/hdfs dfs-rm-r / 123

VI. Expand the content

1. Start and stop a single node (hdfs)

Start Namenode:

$HADOOP_PREFIX/sbin/hadoop-daemon.sh-- config $HADOOP_CONF_DIR-- script hdfs start namenode

Stop Namenode:

$HADOOP_PREFIX/sbin/hadoop-daemon.sh-- config $HADOOP_CONF_DIR-- script hdfs stop namenode

Start Datanode:

$HADOOP_PREFIX/sbin/hadoop-daemons.sh-- config $HADOOP_CONF_DIR-- script hdfs start datanode

Stop Datanode:

HADOOP_PREFIX/sbin/hadoop-daemons.sh-config $HADOOP_CONF_DIR-script hdfs stop datanode

two。 Start and stop a single node (yarn)

Start ResourceManager:

$HADOOP_YARN_HOME/sbin/yarn-daemon.sh-- config $HADOOP_CONF_DIR start resourcemanager

Stop ResourceManager:

$HADOOP_YARN_HOME/sbin/yarn-daemon.sh-- config $HADOOP_CONF_DIR stop resourcemanager

Start NodeManager:

$HADOOP_YARN_HOME/sbin/yarn-daemons.sh-- config $HADOOP_CONF_DIR start nodemanager

Stop NodeManager:

$HADOOP_YARN_HOME/sbin/yarn-daemons.sh-- config $HADOOP_CONF_DIR stop nodemanager

Start WebAppProxy:

$HADOOP_YARN_HOME/sbin/yarn-daemon.sh-- config $HADOOP_CONF_DIR start proxyserver

Stop WebAppProxy:

$HADOOP_YARN_HOME/sbin/yarn-daemon.sh-- config $HADOOP_CONF_DIR stop proxyserver

3. reference

Official website 2.7.2 document URL: http://hadoop.apache.org/docs/stable/

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.