Shulou(Shulou.com)05/31 Report--

This article mainly introduces "how to understand mapreduce wordcount". In daily operation, I believe many people have doubts about how to understand mapreduce wordcount. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the doubts about "how to understand mapreduce wordcount"! Next, please follow the editor to study!

Wordcount counts, can always understand when looking at the code, but the real logic has been unclear, such as map side how to deal with, and how to deal with reduce, now understand.

The principle is that on the map side, each row of data is read and a character in each row is counted, as follows:

Map data {key,value}:

{0,hello word by word}

{1,hello hadoop by hadoop}

The above is the key and value entered on the map side. The following data will be generated after processing on the map side:

{hello,1} {word,1} {by,1} {word,1}

{hello,1} {hadoop,1} {by,1} {hadoop,1}

When you see it, everyone can understand it, but when you see it at the reduce end, you don't understand it, and you don't know how to unify the characters. Through the analysis of the principle of hadoop, it is concluded that the data sent by the map side will be cleaned when it comes to the reduce end. The cleaned data should be the following structure:

[{hello}, {1,1}] [{word}, {1,1}] [{by}, {1,1}] [{hadoop}, {1,1}]

Then input to the reduce end, reduce will loop each values, overlay the data, and output it locally. Please continue to enjoy the specific code, not more than parsing.

Public class WordCount extends Configured implements Tool {

Public static class Map extends Mapper {

Private final static IntWritable one = new IntWritable (1)

Private Text word = new Text ()

Public void map (LongWritable key,Text value, Context context)

Throws IOException,InterruptedException {

String line = value.toString ()

StringTokenizer tokenizer = new StringTokenizer ()

While (tokenizer.hasMoreTokens ()) {

Word.set (tokenizer.nextToken)

Context.write (word,one)

}

}

}

Public static class Reduce extends Reducer {

Public void reduce (Text key,Iterable values,Context context)

Throws IOException,InterruptedException {

Int sum = 0

For (IntWritableval: values) {

Sum + = val.get ()

}

Context.write (key,new IntWritable (sum))

}

}

Public int run (String [] arge) throws Exception {

Job job = new Job (getConf ())

Job.setJarByClass (WordCount.class)

Job.setJobName ("wordcount")

Job.setOutputKeyClass (Text.class)

Job.setOutputValueClass (IntWritable.class)

Job.setMapperClass (Map.class)

Job.setReduceClass (reduce.class)

Job.setInputFormatClass (TextInputFormat.class)

Job.setOutputFormatClass (TextInputFormat.class)

FileInputFormat.setInputPaths (job,new Path (args [0]))

FileInputFormat.setOutputPaths (job, new Path (args [1]))

Boolean success = job.waitForCompletion (true)

Return success? 0: 1

}

Public static void main (String [] args) throws Exception {

Int ret = ToolRunner.run (new WordCount (), args)

System.exit (ret)

}

}

At this point, the study of "how to understand mapreduce wordcount" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.