Shulou(Shulou.com)05/31 Report--

This article mainly explains the "hadoop WordCount case study". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "hadoop WordCount case study" together.

Public class WordCount {

Public static class TokenizerMapper extends Mapper {

Private final static IntWritable one = new IntWritable (1)

Private Text word = new Text ()

/ / the default setting of TextInput is to read a row of data, and the map phase is to split each row read according to our needs.

Public void map (Object key, Text value, Context context) throws IOException, InterruptedException {

StringTokenizer line = new StringTokenizer (value.toString ())

While (line.hasMoreTokens ()) {

Word.set (line.nextToken ())

Context.write (word, one)

}

}

}

/ / in the reduce phase, it is the process of copy the sorted data from the map phase to the reduce task. In this process, a background thread merges the same key value and merges its value into a container similar to a collection. The logic here is that we need to traverse the data in this container, calculate its value, and then output it.

Public static class IntSumReducer extends Reducer {

Private IntWritable result = new IntWritable ()

Public void reduce (Text key, Iterable values, Context context) throws IOException, InterruptedException {

Int sum = 0

For (IntWritableval: values) {

Sum+=val.get ()

}

Result.set (sum)

Context.write (key, result)

}

}

Public static void main (String [] args) throws Exception {

Configuration conf = new Configuration ()

String [] otherArgs = new GenericOptionsParser (conf, args) .getRemainingArgs

If (otherArgs.length! = 2) {

System.err.println ("Usage: wordcount")

System.exit (2)

}

Job job = new Job (conf, "word count")

Job.setJarByClass (WordCount.class)

Job.setMapperClass (TokenizerMapper.class)

Job.setCombinerClass (IntSumReducer.class)

Job.setReducerClass (IntSumReducer.class)

Job.setOutputKeyClass (Text.class)

Job.setOutputValueClass (IntWritable.class)

FileInputFormat.addInputPath (job, new Path (otherArgs [0]))

FileOutputFormat.setOutputPath (job, new Path (otherArgs [1]))

System.exit (job.waitForCompletion (true)? 0: 1)

}

}

Thank you for your reading, the above is the content of "hadoop WordCount case study", after the study of this article, I believe you have a deeper understanding of the problem of hadoop WordCount case study, and the specific use needs to be verified by practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.