Shulou(Shulou.com)05/31 Report--

This article mainly shows you "how to configure the Eclipse-based Hadoop application development environment", the content is easy to understand, clear, hope to help you solve your doubts, the following let the editor lead you to study and learn "how to configure the Eclipse-based Hadoop application development environment" this article.

My development environment:

Operating system centos5.5 one namenode and two datanode

Hadoop version: hadoop-0.20.203.0

Eclipse version: eclipse-java-helios-SR2-linux-gtk.tar.gz (version 3.7 always crashes, depressing)

Step 1: start the hadoop daemon first

For more information, see http://www.cnblogs.com/flyoung2008/archive/2011/11/29/2268302.html

Step 2: install the hadoop plug-in on eclipse

1. Copy the hadoop installation directory / contrib/eclipse-plugin/hadoop-0.20.203.0-eclipse-plugin.jar to the eclipse installation directory / plugins/.

two。 Restart eclipse and configure hadoop installation directory.

If the plug-in is installed successfully, open Window-- > Preferens, and you will find the Hadoop Map/Reduce option, in which you need to configure Hadoop installation directory. Exit when the configuration is complete.

3. Configure Map/Reduce Locations.

Open Map/Reduce Locations in Window-- > Show View.

Create a new Hadoop Location in Map/Reduce Locations. In this View, right-- > New Hadoop Location. In the pop-up dialog box you need to configure Location name, such as Hadoop, as well as Map/Reduce Master and DFS Master. The Host and Port in this are the addresses and ports you configured in mapred-site.xml and core-site.xml respectively. Such as:

Map/Reduce Master

192.168.1.101 9001

DFS Master

192.168.1.101 9000

Exit after configuration. Click DFS Locations-- > Hadoop if the folder (2) shows that the configuration is correct, and if the "connection denied" is displayed, please check your configuration.

Step 3: create a new project.

File-- > New-- > Other-- > Map/Reduce Project

You can take the project name as you like, such as WordCount.

Copy the hadoop installation directory / src/example/org/apache/hadoop/example/WordCount.java to the project you just created.

Step 4: upload the simulation data folder.

In order to run the program, we need an input folder and an output folder.

Create a new word.txt locally

Java C++ python cjava C++ javascript helloworld hadoopmapreduce java hadoop hbase

Create the / tmp/workcount directory on HDFS with the command of hadoop, as follows: bin/hadoop fs-mkdir / tmp/wordcount

Copy the local word.txt to HDFS through the copyFromLocal command: bin/hadoop fs-copyFromLocal / home/grid/word.txt / tmp/wordcount/word.txt

Step 5: run the project

1. In the new project Hadoop, click WordCount.java, and right-click-- > Run As-- > Run Configurations

two。 In the Run Configurations dialog box that pops up, click Java Application, right-- > New, and a new application named WordCount will be created.

3. To configure the running parameters, click Arguments, and enter "the input folder you want to send to the program and the folder where you require the program to save the calculation results" in Program arguments, such as:

Hdfs://centos1:9000/tmp/wordcount/word.txt hdfs://centos1:9000/tmp/wordcount/out

4. If you run Times java.lang.OutOfMemoryError: Java heap space to configure VM arguments (under Program arguments)

-Xms512m-Xmx1024m-XX:MaxPermSize=256m

5. Click Run to run the program.

Click Run, run the program, after a period of time will be run, wait for the end of the run, view the run results, use the command: bin/hadoop fs-ls / tmp/wordcount/out to view the output of the example, found that there are two folders and a file, use the command to view the part-r-00000 file, bin/hadoop fs-cat / tmp/wordcount/out/part-r-00000 can view the results.

C 1 C++ 2 hadoop 2 hbase 1 helloworld 1 java 3 javascript 1 mapreduce 1 python 1 is all the contents of this article entitled "how to configure an Eclipse-based Hadoop Application Development Environment". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.