Shulou(Shulou.com)06/03 Report--

[toc]

Development of Job tool Class for MapReduce

When MapReduce programs write drivers for Mapper and Reducer, there is a lot of repetitive code, so it can be extracted and written into a utility class, which will be used later when writing MapReduce programs.

Job tool class development

The program code is as follows:

Package com.uplooking.bigdata.common.utils;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.mapreduce.*;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import java.io.IOException Public class MapReduceJobUtil {public static Job buildJob (Configuration conf, Class jobClazz, String inputpath, Class mapKeyClass, Class mapValueClass, Path outputpath, Class outkeyClass) Class outvalueClass) throws IOException {String jobName = jobClazz.getSimpleName () Job job = Job.getInstance (conf, jobName); / / set the jar job.setJarByClass (jobClazz) that job runs; / / set the input FileInputFormat.setInputPaths (job, inputpath) of the entire program; job.setInputFormatClass (inputFormat); / / set the parsing class of how to parse the input file into one line of content / / set mapper job.setMapperClass (mapperClass) Job.setMapOutputKeyClass (mapKeyClass); job.setMapOutputValueClass (mapValueClass); / / set the output of the entire program outputpath.getFileSystem (conf) .delete (outputpath, true); / / if the current output directory exists, delete it to avoid .FileAlreadyExistsException FileOutputFormat.setOutputPath (job, outputpath); job.setOutputFormatClass (outputFormat) / / set reducer. If not, there is no need to set if (null! = reducerClass) {job.setReducerClass (reducerClass); job.setOutputKeyClass (outkeyClass); job.setOutputValueClass (outvalueClass);} return job;}}

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.