Shulou(Shulou.com)05/31 Report--

This article mainly explains "how to achieve java map reduce". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "how to achieve java map reduce".

Enter the contents of the file:

An a1

B b2

C c3

D d4

An a1

B b2

C c3

D d4

Output:

An A1 | 0 A1 | 20

B b2 | 5 b2 | 25

C c3 | 10 c3 | 30

D d4 | 15 d4 | 35

Code:

Import java.io.IOException; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.LongWritable;import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat Import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.util.GenericOptionsParser; public class WordCount {public void map (LongWritable key, Text value, Context context) throws IOException, InterruptedException {String [] oriSegs = value.toString () .split ("\ t") String str = oriSegs [1] + "|" + key; context.write (new Text (oriSegs [0]), new Text (str)) }} public static class IntSumReducer extends Reducer {public void reduce (Text key, Iterable values, Context context) throws IOException, InterruptedException {String out = " For (Text val: values) {if (! out.equals ("")) {out + ='\ tasking;} out + = val.toString () } context.write (key, new Text (out));}} public static void main (String [] args) throws Exception {Configuration conf = new Configuration (); conf.set ("mapred.job.queue.name", "platform") String [] otherArgs = new GenericOptionsParser (conf, args). GetRemainingArgs (); if (otherArgs.length! = 2) {System.err.println ("Usage: wordcount"); System.exit (2) } Job job = new Job (conf, "word count"); job.setJarByClass (WordCount.class); job.setMapperClass (TokenizerMapper.class); job.setCombinerClass (IntSumReducer.class); job.setReducerClass (IntSumReducer.class); job.setOutputKeyClass (Text.class) Job.setOutputValueClass (Text.class); job.setNumReduceTasks (1); / / set reducer number FileInputFormat.addInputPath (job, new Path (otherArgs [0])); FileOutputFormat.setOutputPath (job, new Path (otherArgs [1])); System.exit (job.waitForCompletion (true)? 0: 1);}}

Compilation: make.sh compiled to jar file

Javac-classpath / home/hadoop/hadoop-0.20.2-cdh4u0/hadoop-core-0.20.2-cdh4u0.jar:/home/hadoop/hadoop-0.20.2-cdh4u0/lib/commons-cli-1.2.jar-d wordcount_class WordCount.javajar-cvf WordCount.jar-C wordcount_class/.

Perform map reduce task: exec.sh

IN=/user/zhumingliang/tanx_rtb_account/inputOUT=/user/zhumingliang/tanx_rtb_account/output/testhadoop jar WordCount.jar WordCount $IN $OUT

Note:

When entering for a file, the input key of mapper is the character sequence number of the starting position in the file, while the input value of mapper is the entire line.

The input key of reducer is the output of mapper, and the input value of key; reducer is the output value of mapper.

Thank you for your reading, the above is the content of "how to achieve java map reduce", after the study of this article, I believe you have a deeper understanding of how to achieve java map reduce, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.