Shulou(Shulou.com)05/31 Report--

This article introduces the knowledge of "what is the multi-file output of Hadoop and what is the method of customizing file names?" in the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

The first is the class in the output format, that is, job.setOutputFormatClass (... The class in the parameter list:

Public class MoreFileOutputFormat extends Multiple {@ Override protected String generateFileNameForKeyValue (Text key, Text value,Configuration conf) {return "Your name";}}

Here, after inheriting the Multiple class, you must override the generateFileNameForKeyValue () method, which returns a string as the file name of the output file. The content is written by you according to your own needs. At the same time, the values of key and value are also changed according to their own needs.

Next is the code for the Multiple template class:

Import java.io.DataOutputStream;import java.io.IOException;import java.util.HashMap;import java.util.Iterator;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.FSDataOutputStream;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.Writable;import org.apache.hadoop.io.WritableComparable;import org.apache.hadoop.io.compress.CompressionCodec;import org.apache.hadoop.io.compress.GzipCodec;import org.apache.hadoop.mapreduce.OutputCommitter;import org.apache.hadoop.mapreduce.RecordWriter Import org.apache.hadoop.mapreduce.TaskAttemptContext;import org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.util.ReflectionUtils;public abstract class Multiple

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.