How to implement the driver in MapReduce

This article is about how MapReduce implements drivers. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

1. Set the basic properties of job

Job job = new Job ()

Job.setJarByClass (* * .class); / / classes to be executed

Job.setJobName ("job name"); / / name of the job

Number of job.setNumReduce (2); / / reduce

2. Set the classes of Map and Reudce

Job.setMappgerClass (* .class); / / map class

Job.setReduceClass (* .class); / / reduce class

3. Set the input and output format of Job

Void setInputFormatClass (Class theClass)

Void setOutputValueClass (Class theClass)

Void setMapOutputKeyClass (Class theClass)

Void setMapOutputValueClass (Class theClass)

(1) the first two methods set the output of the entire job, that is, the output of reduce. By default, the output type of map is the same as that of reduce, and if the two are not consistent, you need to specify the output type of map through the next two methods.

(2) description of input type: the input type of reduce is determined by the output type of output. The input type of map is determined by the input format. If the input format is FileInputFormat, the input KV type is LongWriterable and Text.

6. Run the program

Job.waitForCompletion ()

We can also set the combine class and the partition class

Job.setCombinerClass (Combine.class)

Job.setPartitionerClass (MyPartition.class)

A picture is attached:

Complete example

Package org.jediael.hadoopdemo.maxtemperature

Import org.apache.hadoop.fs.Path

Import org.apache.hadoop.io.IntWritable

Import org.apache.hadoop.io.Text

Import org.apache.hadoop.mapreduce.Job

Import org.apache.hadoop.mapreduce.lib.input.FileInputFormat

Import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat

Public class MaxTemperature {

Public static void main (String [] args) throws Exception {

If (args.length! = 2) {

System.err

.println ("Usage: MaxTemperature")

System.exit (- 1)

}

/ / 1. Set the basic properties of job

Job job = new Job ()

Job.setJarByClass (MaxTemperature.class)

Job.setJobName ("Max temperature")

/ / 2. Set the classes of Map and Reudce

Job.setMapperClass (MaxTemperatureMapper.class)

Job.setReducerClass (MaxTemperatureReducer.class)

/ / 4. Set the output key type of map and reduce

Job.setOutputKeyClass (Text.class)

Job.setOutputValueClass (IntWritable.class)

/ / 5. Set the input and output path

FileInputFormat.addInputPath (job, new Path (args [0]))

FileOutputFormat.setOutputPath (job, new Path (args [1]))

/ / 6. Run the program

System.exit (job.waitForCompletion (true)? 0: 1)

}

}

Thank you for reading! This is the end of the article on "how to implement the driver in MapReduce". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!