Shulou(Shulou.com)06/01 Report--

This article mainly explains "the use of MultipleOutput examples in Hadoop". The content of the explanation in the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "the use of MultipleOutput examples in Hadoop".

Original data:

Expected results after processing:

MyMapper.java

Package com.xr.text;import java.io.IOException;import org.apache.hadoop.io.LongWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Mapper;public class MyMapper extends Mapper {protected void map (LongWritable key, Text value,Context context) throws IOException, InterruptedException {String [] split = value.toString. Split (";") Context.write (new Text (split [0]), new Text (split [1]));}}

MyReducer.java

Package com.xr.text;import java.io.IOException;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Reducer;import org.apache.hadoop.mapreduce.lib.output.MultipleOutputs;public class MyReducer extends Reducer {private MultipleOutputs mos; / * start before set MultipleOutputs * / protected void setup (Context context) throws IOException, InterruptedException {mos = new MultipleOutputs (context);} protected void reduce (Text K1, Iterable value,Context context) throws IOException, InterruptedException {String key = k1.toString () For (Text t: value) {if ("China" .equals (key)) {mos.write ("china", new Text ("China"), t) } else if ("American" .equals (key)) {mos.write ("usa", new Text ("American"), t);} else if ("Chinese" .equals (key)) {mos.write ("cpeople", new Text ("Chinese"), t) } / * * close MultipleOutputs; * / protected void cleanup (Context context) throws IOException, InterruptedException {mos.close ();}}

JobTest.java

Package com.xr.text;import java.io.IOException;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.FileSystem;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.mapreduce.lib.output.MultipleOutputs;import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat Public class JobTest {public static void main (String [] args) throws IOException, InterruptedException, ClassNotFoundException {String inputPath = "hdfs://192.168.75.100:9000/1.txt"; String outputPath = "hdfs://192.168.75.100:9000/ceshi"; Job job = new Job (); job.setJarByClass (JobTest.class) Job.setMapperClass (MyMapper.class); / * * set MultipleOutput file name * / MultipleOutputs.addNamedOutput (job, "china", TextOutputFormat.class, Text.class, Text.class); MultipleOutputs.addNamedOutput (job, "usa", TextOutputFormat.class, Text.class, Text.class) MultipleOutputs.addNamedOutput (job, "cpeople", TextOutputFormat.class, Text.class, Text.class); job.setReducerClass (MyReducer.class); job.setOutputKeyClass (Text.class); job.setOutputValueClass (Text.class); FileInputFormat.setInputPaths (job, new Path (inputPath)) / / Configuration conf = new Configuration (); / / FileSystem fs = FileSystem.get (conf); / if (fs.exists (new Path (outputPath) {/ / fs.delete (new Path (outputPath), true); / /} FileOutputFormat.setOutputPath (job, new Path (outputPath)) System.exit (job.waitForCompletion (true)? 0: 1);}}

An error is reported during operation:

14-08-12 12:44:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... Using builtin-java classes where applicable

12:44:02 on 14-08-12 ERROR security.UserGroupInformation: PriviledgedActionException as:Xr cause:java.io.IOException: Failed to set permissions of path:\ tmp\ hadoop-Xr\ mapred\ staging\ Xr-1514460710\ .stagi ng to 0700

Exception in thread "main" > java.io.IOException: Failed to set permissions of path:\ tmp\ hadoop-Xr\ mapred\ staging\ Xr-1514460710\ .staging to 0700

At org.apache.hadoop.fs.FileUtil.checkReturnValue (FileUtil.java:689)

At org.apache.hadoop.fs.FileUtil.setPermission (FileUtil.java:662)

At org.apache.hadoop.fs.RawLocalFileSystem.setPermission (RawLocalFileSystem.java:509)

At org.apache.hadoop.fs.RawLocalFileSystem.mkdirs (RawLocalFileSystem.java:344)

At org.apache.hadoop.fs.FilterFileSystem.mkdirs (FilterFileSystem.java:189)

At org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir (JobSubmissionFiles.java:116)

At org.apache.hadoop.mapred.JobClient$2.run (JobClient.java:918)

At org.apache.hadoop.mapred.JobClient$2.run (JobClient.java:912)

At java.security.AccessController.doPrivileged (Native Method)

At javax.security.auth.Subject.doAs (Subject.java:396)

At org.apache.hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java:1149)

At org.apache.hadoop.mapred.JobClient.submitJobInternal (JobClient.java:912)

At org.apache.hadoop.mapreduce.Job.submit (Job.java:500)

At org.apache.hadoop.mapreduce.Job.waitForCompletion (Job.java:530)

At com.xr.text.JobTest.main (JobTest.java:37)

Error solution:

1. Delete the FileUtil.class from hadoop-core-1.1.2.jar.

two。 Then copy / org/apache/hadoop/fs/FileUtil.java from the source code

3. Comment on the checkReturnValue () method

The error is reported again at runtime:

Java.lang.OutOfMemoryError: Java heap space

Solution:

The ok,job was carried out smoothly.

Generate the following files:

Thank you for your reading. The above is the content of "the use of MultipleOutput instances in Hadoop". After the study of this article, I believe you have a deeper understanding of the use of MultipleOutput instances in Hadoop, and the specific usage needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.