Shulou(Shulou.com)06/03 Report--

This article is to share with you about the hadoop MR maven-level code template, the editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

About the use of Maven is no longer verbose, there are many on the Internet, and so many years have not changed much, here only introduce how to build the Hadoop development environment.

1. First create the project

Mvn archetype:generate-DgroupId=my.hadoopstudy-DartifactId=hadoopstudy-DarchetypeArtifactId=maven-archetype-quickstart-DinteractiveMode=false

two。 Then add hadoop's dependent packages hadoop-common, hadoop-client, hadoop-hdfs to the pom.xml file. The added pom.xml file is as follows

4.0.0 my.hadoopstudy hadoopstudy jar 1.0-SNAPSHOT hadoopstudy http://maven.apache.org org.apache.hadoop hadoop-common 2.5.1 org.apache.hadoop hadoop-hdfs 2.5.1 org.apache.hadoop hadoop-client 2.5.1 junit junit 3.8.1 test

3. Test 3.1.First of all, we can test the development of hdfs, assuming that we use the hadoop cluster from the previous Hadoop article, and the class code is as follows

Package my.hadoopstudy.dfs;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.FSDataOutputStream;import org.apache.hadoop.fs.FileStatus;import org.apache.hadoop.fs.FileSystem;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IOUtils;import java.io.InputStream;import java.net.URI;public class Test {public static void main (String [] args) throws Exception {String uri = "hdfs://9.111.254.189:9000/" Configuration config = new Configuration (); FileSystem fs = FileSystem.get (URI.create (uri), config); / / list all files and directories under the / user/fkong/ directory on hdfs [] statuses = fs.listStatus (new Path ("/ user/fkong")); for (FileStatus status: statuses) {System.out.println (status) } / / create a file under the / user/fkong directory of hdfs and write the text FSDataOutputStream os = fs.create (new Path ("/ user/fkong/test.log")); os.write ("Hello World!" .getBytes ()); os.flush (); os.close () / / display the contents of the specified file under / user/fkong of hdfs InputStream is = fs.open (new Path ("/ user/fkong/test.log")); IOUtils.copyBytes (is, System.out, 1024, true);}}

3.2 testing the MapReduce job test code is relatively simple, as follows:

Package my.hadoopstudy.mapreduce;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.Reducer;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.util.GenericOptionsParser;import java.io.IOException Public class EventCount {public static class MyMapper extends Mapper {private final static IntWritable one = new IntWritable (1); private Text event = new Text (); public void map (Object key, Text value, Context context) throws IOException, InterruptedException {int idx = value.toString (). IndexOf (""); if (idx > 0) {String e = value.toString (). Substring (0, idx); event.set (e); context.write (event, one) } public static class MyReducer extends Reducer {private IntWritable result = new IntWritable (); public void reduce (Text key, Iterable values, Context context) throws IOException, InterruptedException {int sum = 0; for (IntWritableval: values) {sum + = val.get ();} result.set (sum); context.write (key, result) } public static void main (String [] args) throws Exception {Configuration conf = new Configuration (); String [] otherArgs = new GenericOptionsParser (conf, args). GetRemainingArgs (); if (otherArgs.length < 2) {System.err.println ("Usage: EventCount"); System.exit (2);} Job job = Job.getInstance (conf, "event count"); job.setJarByClass (EventCount.class); job.setMapperClass (MyMapper.class) Job.setCombinerClass (MyReducer.class); job.setReducerClass (MyReducer.class); job.setOutputKeyClass (Text.class); job.setOutputValueClass (IntWritable.class); FileInputFormat.addInputPath (job, new Path (otherArgs [0])); FileOutputFormat.setOutputPath (job, new Path (otherArgs [1])); System.exit (job.waitForCompletion (true)? 0: 1);}}

Run the "mvn package" command to generate the jar package hadoopstudy-1.0-SNAPSHOT.jar, and copy the jar file to the hadoop installation directory

Let's assume that we need to analyze the Event information in several log files to count the number of Event, so create directories and files

/ tmp/input/event.log.1/tmp/input/event.log.2/tmp/input/event.log.3

Because there is only a column to be done here, the contents of each file can be the same, if the contents are as follows

JOB_NEW... JOB_NEW... JOB_FINISH... JOB_NEW... JOB_FINISH...

Then copy these files to HDFS

$bin/hdfs dfs-put / tmp/input / user/fkong/input

Run a mapreduce job

$bin/hadoop jar hadoopstudy-1.0-SNAPSHOT.jar my.hadoopstudy.mapreduce.EventCount / user/fkong/input / user/fkong/output

View the execution result

The above $bin/hdfs dfs-cat / user/fkong/output/part-r-00000 is what the hadoop MR maven-level code template looks like, and the editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.