Shulou(Shulou.com)06/01 Report--

This article introduces the knowledge of "how to use Maven to create a Hadoop2 project". Many people will encounter this dilemma in the operation of actual cases, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Create a Hadoop2 project using Maven

Jdk.tools jdk.tools 1.7 system Java_Home/lib/tools.jar org.apache.hadoop hadoop-common 2.5.2 jdk.tools jdk.tools Org.apache.hadoop hadoop-hdfs 2.5.2 org.apache.hadoop hadoop-client 2.5.2

Connect to the Hadoop2 environment, for example:

Import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.conf.Configuration;public class IPStarter {public static void main (String [] args) throws Exception {Configuration conf = new Configuration () / / Connect to hadoop2 cluster debugging conf.set ("fs.defaultFS", "hdfs://localhost:9000"); conf.set ("mapreduce.framework.name", "yarn"); conf.set ("yarn.resourcemanager.address", "localhost:8032"); conf.set ("yarn.resourcemanager.scheduler.address", "localhost:8030") / / conf.set ("mapred.remote.os", "Linux"); Job job = new Job (conf, "JobName"); job.setJar ("/ home/grid/study/hadoopDev2/target/hadoopDev2-1.0-SNAPSHOT.jar"); / / use job.setMapperClass (IPMap.class) when debugging the development environment; job.setReducerClass (IPReduce.class); job.setMapOutputKeyClass (Text.class) Job.setMapOutputValueClass (Text.class); job.setOutputKeyClass (Text.class); job.setOutputValueClass (IntWritable.class); FileInputFormat.addInputPath (job, new Path ("hdfs://localhost:9000/dev2/in/access.20120104.log")); FileOutputFormat.setOutputPath (job, new Path ("hdfs://localhost:9000/dev2/out/ip")); System.exit (job.waitForCompletion (true)? 0: 1);}}

PS: copy the HADOOP_HOME/etc/hadoop/log4j.properties to the program's src directory so that you can see the detailed log of Job execution on the console when debugging.

This is the end of "how to use Maven to create a Hadoop2 project". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.