Learning log-partitioner and sampler

In Mapreduce:

The shuffle phase is between map and reduce, and you can customize sorting, custom partitions, and custom grouping!

In Mapreduce, the data from map is a key-value pair. By default, hashPatitionner is used to partition the data from map.

There are several other ways to partition:

RandomSampler sampler = new InputSampler.RandomSampler (3000, 3000, 10); IntervalSampler sampler2 = new InputSampler.IntervalSampler (0.333, 10); SplitSampler sampler3 = new InputSampler.SplitSampler (reduceNumber)

Implementation and details

Public class TotalSortMR {@ SuppressWarnings ("deprecation") public static int runTotalSortJob (String [] args) throws Exception {Path inputPath = new Path (args [0]); Path outputPath = new Path (args [1]); Path partitionFile = new Path (args [2]); int reduceNumber = Integer.parseInt (args [3]) / / three samplers RandomSampler sampler = new InputSampler.RandomSampler (1, 3000, 10); IntervalSampler sampler2 = new InputSampler.IntervalSampler (0.333, 10); SplitSampler sampler3 = new InputSampler.SplitSampler (reduceNumber); / / Task initialization Configuration conf = new Configuration (); Job job = Job.getInstance (conf); job.setJobName ("Total-Sort") Job.setJarByClass (TotalSortMR.class); job.setInputFormatClass (KeyValueTextInputFormat.class); job.setMapOutputKeyClass (Text.class); job.setMapOutputValueClass (Text.class); job.setNumReduceTasks (reduceNumber); / / set all partition classes job.setPartitionerClass (TotalOrderPartitioner.class); / / partition files referenced by partition classes TotalOrderPartitioner.setPartitionFile (conf, partitionFile) / / which sampler InputSampler.writePartitionFile (job, sampler) is used in the partition; / / the input and output paths of job are FileInputFormat.setInputPaths (job, inputPath); FileOutputFormat.setOutputPath (job, outputPath); outputPath.getFileSystem (conf) .delete (outputPath, true); return job.waitForCompletion (true)? 0: 1 } public static void main (String [] args) throws Exception {System.exit (runTotalSortJob (args));}}

The default input format for job is TextInputFormat, which is in the form of key-value, key is the line label of each line, and value is the content of each line. Can be changed

Job.setInputFormatClass (,....)

Generally, you need to set the output format of mapper for later use.