Shulou(Shulou.com)05/31 Report--

This article mainly explains "how to solve the Iterable parameter problem of reduce in Mapreduce program". The content of the explanation in this article is simple and clear, and it is easy to learn and understand. next, please follow the editor's train of thought to slowly deepen, to study and learn "how to solve the Iterable parameter problem of reduce in Mapreduce program".

Today, when iterating over the parameter Iterable of reduce, we found a problem, that is, the next () method of Iterator returns the same object each time, and next () only modifies the value of the Writable object instead of returning a new Writable object.

Use wordcount to verify:

My code is as follows:

Protected void reduce (Text key, Iterable values, Reducer.Context context) throws IOException, InterruptedException {int sum = 0; / / Save each IntWritable to list List intWritables = new ArrayList (); for (IntWritableval: values) {intWritables.add (val); sum + = val.get () } if (intWritables.size () > 1) {/ / when list size is greater than 1, verify whether the first element and the second element are the same object System.out.println ("objects is same->" + (intWritables.get (0) = = intWritables.get (1));} result.set (sum); context.write (key, result);}

Log output:

Objects is same-> true

The implementation of this Iterable is org.apache.hadoop.mapreduce.task.ReduceContextImpl.ValueIterable

The Iterator implementation is org.apache.hadoop.mapreduce.task.ReduceContextImpl.ValueIterator

When next () is implemented, the deserialize (Writable w) method of org.apache.hadoop.io.serializer.WritableSerialization is called.

Writable deserialize (Writable w) IOException {Writable writable; (w = =) {writable = (Writable) ReflectionUtils. (, getConf ());} {writable = w;} writable.readFields (); writable;}

This method simply calls the readFields method of the input parameter w and does not create a new object, unless w is null

Thank you for your reading, the above is the content of "how to solve the Iterable parameter problem of reduce in Mapreduce program". After the study of this article, I believe you have a deeper understanding of how to solve the problem of Iterable parameter problem of reduce in Mapreduce program. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.