Shulou(Shulou.com)05/31 Report--

This article mainly introduces "how to realize the minimum or maximum temperature of the weather station by MapReduce". In the daily operation, I believe many people have doubts about how to calculate the minimum or maximum temperature of the weather station by MapReduce. The editor consulted all kinds of data and sorted out simple and easy-to-use operation methods. I hope it will be helpful for everyone to answer the doubt of "how to realize the minimum or maximum temperature of the weather station by MapReduce". Next, please follow the editor to study!

TemperatureMR.java

Package cn.kissoft.hadoop.week05;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat Public class TemperatureMR {public static void main (String [] args) throws Exception {if (args.length! = 2) {System.err.println ("Usage: Temperature"); System.exit (- 1);} Job job = new Job (); job.setJarByClass (TemperatureMR.class); job.setJobName ("Max and min temperature") FileInputFormat.addInputPath (job, new Path (args [0])); FileOutputFormat.setOutputPath (job, new Path (args [1])); job.setMapperClass (TemperatureMapper.class); job.setReducerClass (TemperatureReducer.class); job.setOutputKeyClass (Text.class); job.setOutputValueClass (IntWritable.class); System.exit (job.waitForCompletion (true)? 0: 1);}}

TemperatureMapper.java

Package cn.kissoft.hadoop.week05;import java.io.IOException;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.LongWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Mapper;public class TemperatureMapper extends Mapper {private static final int MISSING = 9999; @ Override public void map (LongWritable key, Text value, Context context) throws IOException, InterruptedException {String line = value.toString () String year = line.substring (0,4); int airTemperature = Integer.parseInt (line.substring (13,19). Trim (); if (Math.abs (airTemperature)! = MISSING) {context.write (new Text (year), new IntWritable (airTemperature));}

MaxTemperatureReducer.java

Package cn.kissoft.hadoop.week05;import java.io.IOException;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Reducer;public class TemperatureReducer extends Reducer {@ Override public void reduce (Text key, Iterable values, Context context) throws IOException, InterruptedException {int maxValue = Integer.MIN_VALUE; int minValue = Integer.MAX_VALUE For (IntWritable value: values) {maxValue = Math.max (maxValue, value.get ()); minValue = Math.min (minValue, value.get ());} context.write (key, new IntWritable (maxValue)); context.write (key, new IntWritable (minValue));}}

Running process

[wukong@bd11 guide] $hadoop jar pc.jar cn.kissoft.hadoop.week05.TemperatureMR. / ch02/1959.txt. / ch02/out/Warning: $HADOOP_HOME is deprecated.14/08/15 16:29:32 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.14/08/15 16:29:32 INFO input.FileInputFormat: Total input paths to process: 114-08-15 16:29:32 INFO util.NativeCodeLoader: Loaded the native-hadoop library14/08/15 16:29:32 WARN snappy.LoadSnappy: Snappy native library not loaded14/08/15 16:29:34 INFO mapred.JobClient: Running job: job_201408151617_000314/08/15 16:29:35 INFO mapred.JobClient: map 0% reduce 0Then 08 reduce 15 16:29:47 INFO mapred.JobClient: map reduce 08 reduce 15 16:30:00 INFO mapred.JobClient: map 100 reduce 100-08-15 16:30:04 INFO mapred.JobClient: Job complete: job_201408151617_000314/08/15 16:30:04 INFO mapred.JobClient: Counters: 2914-08-15 16:30:04 INFO mapred.JobClient: Job Counters 14-08-15 16:30:04 INFO mapred.JobClient: Launched reduce tasks=114/08/15 16:30:04 INFO mapred.JobClient : SLOTS_MILLIS_MAPS=1498914/08/15 16:30:04 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms) = 014 SLOTS_MILLIS_MAPS=1498914/08/15 16:30:04 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms) = 014 INFO mapred.JobClient 16:30:04 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms) = 014 INFO mapred.JobClient 15 16:30:04 INFO mapred.JobClient: Launched map tasks=114/08/15 16:30:04 INFO mapred.JobClient: Data-local map tasks=114/08/15 16:30:04 INFO mapred. JobClient: SLOTS_MILLIS_REDUCES=1282514/08/15 16:30:04 INFO mapred.JobClient: File Output Format Counters14/08/15 16:30:04 INFO mapred.JobClient: Bytes Written=1914/08/15 16:30:04 INFO mapred.JobClient: FileSystemCounters14/08/15 16:30:04 INFO mapred.JobClient: FILE_BYTES_READ=918048614/08/15 16:30:04 INFO mapred.JobClient: HDFS_BYTES_READ=2754447514/08/15 16:30:04 INFO mapred.JobClient: FILE_BYTES_WRITTEN=1388690814/08/15 16:30:04 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=1914/08/15 16:30:04 INFO mapred.JobClient: File Input Format Counters 14-08-15 16:30:04 INFO mapred.JobClient: Bytes Read=2754436814/08/15 16:30:04 INFO mapred.JobClient: Map-Reduce Framework14/08/15 16:30:04 INFO mapred.JobClient: Map output materialized bytes=459024014/08/15 16:30:04 INFO mapred.JobClient: Map input records=44426414/08/15 16:30:04 INFO mapred.JobClient: Reduce shuffle bytes=459024014/08/15 16:30:04 INFO mapred.JobClient: Spilled Records=125188214/08/15 16:30:04 INFO mapred.JobClient: Map output bytes=375564614/08/15 16:30:04 INFO mapred.JobClient: Total committed heap usage (bytes) = 21886566414 Spilled Records=125188214/08/15 15 16:30:04 INFO mapred.JobClient: CPU time spent (ms) = 628014 16:30:04 INFO mapred.JobClient: Combine input records=014/08/15 16:30:04 INFO mapred.JobClient: SPLIT_RAW_BYTES=10714/08/15 16:30:04 INFO mapred.JobClient: Reduce input records=41729414/08/15 16:30:04 INFO mapred.JobClient: Reduce input groups=114/08/15 16:30:04 INFO mapred.JobClient: Combine output records=014/08/15 16:30:04 INFO mapred.JobClient: Physical memory (bytes) snapshot=32298598414/08/15 16:30:04 INFO mapred.JobClient: Reduce output records=214/08/15 16:30:04 INFO mapred.JobClient: Virtual memory (bytes) snapshot=145557913614/08/15 16:30:04 INFO mapred.JobClient: Map output records=417294

Running result

[wukong@bd11 guide] $hadoop fs-cat. / ch02/out/part-r-00000Warning: $HADOOP_HOME is deprecated.1959 4181959-400

Screenshot

At this point, the study on "how to achieve the minimum or maximum temperature of the weather station by MapReduce" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.