Shulou(Shulou.com)05/31 Report--

This article introduces the relevant knowledge of "how to use MapReduce to find the total salary of various departments". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

data

EMPNO ENAME JOB MGR HIREDATE SAL COMM DEPTNO

7369 SMITH CLERK 7902 17-December-80800 20 7499 ALLEN SALESMAN 7698 20-February-81 1600 1600 30 7521 WARD SALESMAN 7698 22-February-81 1250 1250 30 7566 JONES MANAGER 7839 02-April-81 2975 20 7654 MARTIN SALESMAN 7698 28-September-81 1250 1400 30 7698 BLAKE MANAGER 7839 01-May-81 2850 30 7782 CLARK MANAGER 7839 09-June-81 2450 10 7839 KING PRESIDENT 17-November-81 5000 10 7844 TURNER SALESMAN 7698 08-September-81 1500 0 30 7900 JAMES CLERK 7698 03-December-81 950 30 7902 FORD ANALYST 7566 03-December- 81 3000 20 7934 MILLER CLERK 7782 23-January-82 1300 10 Code package cn.kissoft.hadoop.week07 Import java.io.IOException;import java.text.DateFormat;import java.text.SimpleDateFormat;import java.util.Date;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.conf.Configured;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.LongWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.Reducer Import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;import org.apache.hadoop.util.Tool;import org.apache.hadoop.util.ToolRunner;import cn.kissoft.hadoop.util.HdfsUtil / * Homework-01: calculate the total salary of each department * * @ author wukong (jinsong.sun@139.com) * / public class TotalSalaryByDeptMR extends Configured implements Tool {public static class M extends Mapper {@ Override public void map (LongWritable key, Text value, Context context) throws IOException, InterruptedException {String line = value.toString () String deptno = line.substring (79). Trim (); String sal = line.substring (57, 68). Trim (); int salary = Integer.valueOf (sal); context.write (new Text (deptno), new IntWritable (salary)) }} public static class R extends Reducer {@ Override public void reduce (Text key, Iterable values, Context context) throws IOException, InterruptedException {int sum = 0 For (IntWritable value: values) {sum + = value.get ();} context.write (key, new IntWritable (sum)) } @ Override public int run (String [] args) throws Exception {Configuration conf = getConf (); Job job = new Job (conf, "Job-TotalSalaryByDeptMR"); / / job.setJarByClass (this.getClass ()); job.setMapperClass (M.class) Job.setMapOutputKeyClass (Text.class); job.setMapOutputValueClass (IntWritable.class); job.setReducerClass (R.class); job.setOutputFormatClass (TextOutputFormat.class); / / job.setOutputKeyClass (NullWritable.class); / / specify the format job.setOutputKeyClass (Text.class) of the output KEY / / specify the format of the output KEY job.setOutputValueClass (IntWritable.class); / / specify the format of the output VALUE FileInputFormat.addInputPath (job, new Path (args [0])); / / input path FileOutputFormat.setOutputPath (job, new Path (args [1])) / / output path return job.waitForCompletion (true)? 0: 1; / / job.waitForCompletion (true); / / return job.isSuccessful ()? 0: 1 } / * * @ param args hdfs://bd11:9000/user/wukong/w07/emp.txt hdfs://bd11:9000/user/wukong/w07/out01/ * @ throws Exception * / public static void main (String [] args) throws Exception {checkArgs (args); HdfsUtil.rm (args [1], true) Date start = new Date (); int res = ToolRunner.run (new Configuration (), new TotalSalaryByDeptMR (), args); printExcuteTime (start, new Date ()); System.exit (res);} / * to determine whether the number of parameters is correct, and if there are no parameters to run, it will be displayed for program description. * * @ param args * / private static void checkArgs (String [] args) {if (args.length! = 2) {System.err.println (""); System.err.println ("Usage: Test_1

"); System.err.println (" Example: hadoop jar ~ / Test_1.jar hdfs://localhost:9000/home/james/Test_1 hdfs://localhost:9000/home/james/output "); System.err.println (" Counter: ") System.err.println ("\ t" + "LINESKIP" + "\ t" + "Lines which are too short"); System.exit (- 1) Run time of the print program * * @ param start * @ param end * / private static void printExcuteTime (Date start, Date end) {DateFormat formatter = new SimpleDateFormat ("yyyy-MM-dd HH:mm:ss") Float time = (float) ((end.getTime ()-start.getTime ()) / 60000.0); System.out.println ("Task start:" + formatter.format (start)); System.out.println ("Task end:" + formatter.format (end) System.out.println ("Task time:" + String.valueOf (time) + "minutes");}} run results 10 875020 677530 9400 console 23:01:01 on 14-08-31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... Using builtin-java classes where applicable14/08/31 23:01:01 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf (Class) or JobConf#setJar (String). 14-08-31 23:01:01 INFO input.FileInputFormat: Total input paths to process: 114-08-31 23:01:02 WARN snappy.LoadSnappy: Snappy native library not loaded14/08/31 23:01:02 INFO mapred.JobClient: Running job: job_local248108448_000114/08/31 23:01:02 INFO mapred.LocalJobRunner: Waiting for map tasks14/08/31 23:01:02 INFO mapred.LocalJobRunner: Starting task: attempt_local248108448_0001_m_000000 _ 014Accord 08 INFO mapred.Task 31 23:01:02 INFO mapred.Task: Using ResourceCalculatorPlugin: null14/08/31 23:01:02 INFO mapred.MapTask: Processing split: hdfs://bd11:9000/user/wukong/w07/emp.txt:0+111914/08/31 23:01:02 INFO mapred.MapTask: io.sort.mb = 10014 INFO mapred.MapTask 31 23:01:02 INFO mapred.MapTask: data buffer = 79691776 INFO mapred.MapTask 31 23:01:02 INFO mapred.MapTask: record buffer = 262144 INFO mapred.MapTask 3276801408 / 31 23:01:02 INFO mapred.MapTask: Starting flush of map output14/08/31 23:01:02 INFO mapred.MapTask: Finished spill 014/08/31 23:01:02 INFO mapred.Task: Task:attempt_local248108448_0001_m_000000_0 is done. And is in the process of commiting14/08/31 23:01:02 INFO mapred.LocalJobRunner: 14-08-31 23:01:02 INFO mapred.Task: Task 'attempt_local248108448_0001_m_000000_0' done.14/08/31 23:01:02 INFO mapred.LocalJobRunner: Finishing task: attempt_local248108448_0001_m_000000_014/08/31 23:01:02 INFO mapred.LocalJobRunner: Map task executor complete.14/08/31 23:01:02 INFO mapred.Task: Using ResourceCalculatorPlugin: null14 / 08/31 23:01:02 INFO mapred.LocalJobRunner: 14-08-31 23:01:02 INFO mapred.Merger: Merging 1 sorted segments14/08/31 23:01:02 INFO mapred.Merger: Down to the last merge-pass With 1 segments left of total size: 110 bytes14/08/31 23:01:02 INFO mapred.LocalJobRunner: 14-08-31 23:01:02 INFO mapred.Task: Task:attempt_local248108448_0001_r_000000_0 is done. And is in the process of commiting14/08/31 23:01:02 INFO mapred.LocalJobRunner: 14-08-31 23:01:02 INFO mapred.Task: Task attempt_local248108448_0001_r_000000_0 is allowed to commit now14/08/31 23:01:02 INFO output.FileOutputCommitter: Saved output of task 'attempt_local248108448_0001_r_000000_0' to hdfs://bd11:9000/user/wukong/w07/out0114/08/31 23:01:02 INFO mapred.LocalJobRunner: reduce > reduce14/08/ 31 23:01:02 INFO mapred.Task: Task 'attempt_local248108448_0001_r_000000_0' done.14/08/31 23:01:03 INFO mapred.JobClient: map 100 reduce 100-08-31 23:01:03 INFO mapred.JobClient: Job complete: job_local248108448_000114/08/31 23:01:03 INFO mapred.JobClient: Counters: 1914-08-31 23:01:03 INFO mapred.JobClient: File Output Format Counters 14-08-31 23:01:03 INFO mapred.JobClient : Bytes Written=2414/08/31 23:01:03 INFO mapred.JobClient: File Input Format Counters14/08/31 23:01:03 INFO mapred.JobClient: Bytes Read=111914/08/31 23:01:03 INFO mapred.JobClient: FileSystemCounters14/08/31 23:01:03 INFO mapred.JobClient: FILE_BYTES_READ=42614/08/31 23:01:03 INFO mapred.JobClient: HDFS_BYTES_READ=223814/08/31 23:01:03 INFO mapred.JobClient: FILE_BYTES _ WRITTEN=13857814/08/31 23:01:03 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=2414/08/31 23:01:03 INFO mapred.JobClient: Map-Reduce Framework14/08/31 23:01:03 INFO mapred.JobClient: Reduce input groups=314/08/31 23:01:03 INFO mapred.JobClient: Map output materialized bytes=11414/08/31 23:01:03 INFO mapred.JobClient: Combine output records=014/08/31 23:01:03 INFO mapred.JobClient: Map input records=1214/ 08Total committed heap usage 31 23:01:03 INFO mapred.JobClient: Reduce shuffle bytes=014/08/31 23:01:03 INFO mapred.JobClient: Reduce output records=314/08/31 23:01:03 INFO mapred.JobClient: Spilled Records=2414/08/31 23:01:03 INFO mapred.JobClient: Map output bytes=8414/08/31 23:01:03 INFO mapred.JobClient: Total committed heap usage (bytes) = 32610713614 23:01:03 INFO mapred.JobClient: SPLIT_RAW_BYTES=10514/ 23:01:03 INFO mapred.JobClient: Map output records=1214/08/31 23:01:03 INFO mapred.JobClient: Combine input records=014/08/31 23:01:03 INFO mapred.JobClient: Reduce input records=12 Task start: 2014-08-31 23:01:01 Task end: 2014-08-31 23:01:03 Task time: 0.024416666 minutes "how to use MapReduce to calculate the total salary of various departments" this is the end of the introduction. Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.