Shulou(Shulou.com)06/01 Report--

This article introduces the knowledge about "how to build Hadoop 2.7.1 distributed cluster for VMware virtual machine". In the actual case operation process, many people will encounter such difficulties. Next, let Xiaobian lead you to learn how to deal with these situations! I hope you can read carefully and learn something!

[Architecture introduction]

[Preparations]

1. VMware: 11.0.0 build-2305329 (download: pan.baidu.com/s/1skq0oP3)

2. CentOS: CentOS-6.7-x86_64-bin-DVD1.iso (download: pan.baidu.com/s/1o7xYgGE)

or Ubuntu(CentOS for example, CentOS for general production environments)

3. Local machine configuration:

5. JDK

(1) Version 1.7.9

(2) Download address: www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html

[Run MapReduce]

1. prepare data

Create a wordcount.txt file in your local directory and type in English, such as:

aa bb cc dd

2. upload files

hadoop fs -put ./ wordcount.txt /workspace

3. execution command

hadoop jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar wordcount /workspace/wordcount.txt /outpath

4. UI View Task

http://192.168.159.130:8088/cluster/app/RUNNING

5. view results

hadoop fs -cat /outpath/*

aa 1

bb 1

cc 1

dd 1

[Problems encountered]

1. reject the connection

Problem Description: Problems such as connection rejection during startup and running tasks.

Solution: Check that the firewall is turned off.

2. resource manager failed to start

Problem description: When starting the cluster, other processes are normal, only the resource manager fails to start, or fails after only a few seconds.

Solution: Start the cluster by executing sh start-all.sh on the hadoop02 machine (where the resourcemanager resides).

3. Exception executing start-all.sh

Error:Cannot find configuration directory:/etc/hadoop

Solution: Replace export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"etc/hadoop"} with export HADOOP_CONF_DIR=/usr/hadoop-2.6.0/etc/hadoop in the hadoop-env.sh configuration file and run source hadoop-env.sh to make it take effect immediately!

4. Run Wordcount error

Problem Description: When running with WordCount, mapreduce_shuffle does not exist.

Solution: Add the following configuration to yarn-site.xml

yarn.nodemanager.aux-services

mapreduce_shuffle

yarn.nodemanager.aux-services.mapreduce.shuffle.class

org.apache.hadoop.mapred.ShuffleHandler

5. Error after running Wordcount for a while

Hadoop org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container

Solution: Baidu said that the cluster time is not synchronized, solution reference:

http://www.chinadmd.com/file/tixiarput3eweszaceuzpttc_1.html

http://www.aboutyun.com/thread-11397-1-2.html

http://www.linuxde.net/2013/02/12232.html

"VMware virtual machine how to build Hadoop 2.7.1 distributed cluster" content introduced here, thank you for reading. If you want to know more about industry-related knowledge, you can pay attention to the website. Xiaobian will output more high-quality practical articles for everyone!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.