Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to configure hadoop for you. The editor thinks it is very practical, so I share it with you as a reference. I hope you can get something after reading this article.

1. Configure hadoop

First, go to github to download the source code: http://hadoop.apache.org/releases.html#News

In linux to install jdk and configure ssh password-free login, in fact, at the beginning I did not configure the zone password-free login, and later found that password-free login is really convenient. To log in without a password is to enter: ssh-keygen on the console and then enter all the way.

I unzipped hadoop and put it in / home/admin1/ download / hadoop-2.5.2. We mainly configure files in etc/hadoop.

1 、 core-site.xml

Fs.defaultFS hdfs://ubuntu2:9000 hadoop.tmp.dir / home/admin1/hadoop/hadoop-2.5.2/tmp/hadoop

2 、 hdfs-site.xml

Dfs.replication 1

3 、 mapred-site.xml

Mapreduce.framework.name yarn

4. In yarn-site.xml

Yarn.resourcemanager.hostname ubuntu2 yarn.nodemanager.aux-services mapreduce_shuffle

5. Add to hadoop-env.sh

Export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-amd64# The jsvc implementation to use. Jsvc is required to run secure datanodes.#export JSVC_HOME=$ {JSVC_HOME} export HADOOP_CONF_DIR=$ {HADOOP_CONF_DIR:- "/ home/admin1/hadoop/hadoop-2.5.2"}

6. Remember to change salver to the name of your host. Here I change localhost to ubuntu2. Because my host name is ubuntu2.

You also need to configure java_home: in hadoop-env.sh: (just change to your absolute path, not relative path).

Export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-amd64

If you run an error, it should be that java_home is not configured

Enter: sudo gedit / etc/profile from the command line

Add at the end:

JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-amd64PATH=$JAVA_HOME/bin:$PATHCLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarexport JAVA_HOMEexport PATHexport CLASSPATH

This should basically work:

Startup method: used in / home/admin1/ download / hadoop-2.5.2:

Bin/hadoop namenode-format

Sbin/start-all.sh

The access address is:

Http://localhost:50070/

Http://localhost:8088/cluster

If you want to close it: sbin/stop-all.sh

Question:

If hadoop reports an error, configure the hadoop command:

Export HADOOP_HOME=/home/admin1/ download / hadoop-2.5.2

Export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

If datanode is not started

Rm-rf / usr/hadoop/tmp/*

Rm-rf / tmp/hadoop*

Do the above two steps before 1.

2. Configure eclipse

Because I installed eclipse directly in linux, and I used version 4.5, there were all kinds of weird problems. At first I thought there was something wrong with hadoop-eclipse-plugin-2.5.2.jar, and then I went to recompile it with ant and hadoop2x-eclipse-plugin-master, and the compilation method looks something like this:

1. Download hadoop2x-eclipse-plugin-master, https://github.com/winghc/hadoop2x-eclipse-plugin first.

2. Download ant: http://ant.apache.org/bindownload.cgi

3. Decompress it separately. If you are under win, you need to configure the environment variables. If you do not need to configure it under linux, you can actually run it. Enter the / hadoop2x-eclipse-plugin-master/src/contrib/eclipse-plugin you just unzipped.

4. In which directory above, modify the following files:

In vxy: libraries.properties

The version inside remember to correspond to your direct version, if you can not find your version you can go to which hadoop file share/hadoop to find, in fact, there is a very simple way, you can directly copy my code in the past, wait for compilation time to report which jar package version is wrong, and then you can correct it according to the error prompt.

# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License.# You may obtain a copy of the License at## http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "ASIS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License. # This properties file lists the versions of the various artifacts used by hadoop and components.#It drives ivy and the generation of a maven POM# This is the version of hadoop we are generatinghadoop.version=2.5.2hadoop-gpl-compression.version=0.1.0 # These are the versions of our dependencies (in alphabetical order) apacheant.version=1.7.0ant-task.version=2.0.10 asm.version=3.2aspectj.version=1.6.5aspectj.version=1.6.11 checkstyle.version=4.2 commons-cli.version=1.2commons-codec.version=1.4commons-collections. Version=3.2.1commons-configuration.version=1.6commons-daemon.version=1.0.13commons-httpclient.version=3.1commons-lang.version=2.6commons-logging.version=1.1.3commons-logging-api.version=1.0.4commons-math.version=3.1.1commons-el.version=1.0commons-fileupload.version=1.2commons-io.version=2.4commons-net.version=3.1core.version=3.1.1coreplugin.version=1.3.2 hsqldb.version=1.8.0.10 ivy.version=2.1.0 jasper.version=5.5.12jackson.version=1.9 .13#not able to figureout the version of jsp & jsp-api version to get it resolved throught ivy# but still declared here as we are going to have a local copy from the lib folderjsp.version=2.1jsp-api.version=5.5.12jsp-api-2.1.version=6.1.14jsp-2.1.version=6.1.14jets3t.version=0.6.1jetty.version=6.1.26jetty-util.version=6.1.26jersey-core.version=1.9jersey-json.version=1.9jersey-server.version=1.9junit.version=4.11jdeb.version=0.8jdiff .version = 1.0.9json.version=1.0 kfs.version=0.1 log4j.version=1.2.17lucene-core.version=2.3.1 mockito-all.version=1.8.5jsch.version=0.1.42 oro.version=2.0.8 rats-lib.version=0.5.1 servlet.version=4.0.6servlet-api.version=2.5slf4j-api.version=1.7.5slf4j-log4j12.version=1.7.5 wagon-http.version=1.0-beta-2xmlenc.version=0.52xerces.version=1.4.4protobuf.version=2.5.0guava.version=11.0.2netty.version=3.6.2.Final

Build.xml

This is also the same, if your version is not correct, you can follow the above reason. Remember to write at the top of the file header of xml.

MakePlus.sh

Ant jar-Dversion=2.5.2-Declipse.home=/home/admin1/Public/eclipse-Dhadoop.home=/home/admin1/ download / hadoop-2.5.2

Description: Declipse is your eclipse installation directory, Dhadoop is your hadoop installation directory.

Then run. / makePlus.sh and you can compile. After compiling, put it into the plugins of eclipse, and then restart eclipse.

Well, speaking of which, my heart broke down, is there any complication here, and in the end, I restarted eclipse unexpectedly without success. Later, I found that it was the reason for eclipse for so long. I simply had 10,000 heads in my heart. Later, I deleted version 4.5 and changed it to 4.4. Of course, there is a small episode in the middle, that is, you can't right-click New Hadoop after configuration, so I typed it on the command line:

. / eclipse-clean-consolelog-debug

Then restart it and there will be no problem.

Now let's look at the running configuration:

You can see some of the folders I just created on the browser:

This is the end of the article on "how to configure hadoop". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.