Shulou(Shulou.com)06/01 Report--

This article mainly introduces how hadoop avoids data tilt through cachefile, which has certain reference value, which can be used for reference by interested friends. I hope you will gain a lot after reading this article.

Package hello_hadoop;import java.io.BufferedReader;import java.io.BufferedWriter;import java.io.File;import java.io.FileInputStream;import java.io.FileReader;import java.io.FileWriter;import java.io.IOException;import java.io.InputStream;import java.io.InputStreamReader;import java.net.URI;import java.net.URISyntaxException;import org.apache.commons.logging.Log;import org.apache.commons.logging.LogFactory;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.filecache.DistributedCache Import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.LongWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.input.FileSplit;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat Public class GetFileName {/ / get the file name processed, and cache the required files to the corresponding node private final static Log LOG = LogFactory.getLog (GetFileName.class); public static void main (String [] args) throws IOException, ClassNotFoundException, InterruptedException, URISyntaxException {LOG.info ("Go into the main method."); Configuration conf = new Configuration () Job job = Job.getInstance (conf); job.setJarByClass (GetFileName.class); job.setMapOutputKeyClass (Text.class); job.setMapOutputValueClass (Text.class); job.setMapperClass (GetFileNameMapper.class); job.setNumReduceTasks (0) / / add the description of DistributedCache data134:9000namenode on the cluster # the link to the thelinkofthefile modification file. You need to use job.addCacheFile (new URI ("hdfs://data134:9000/home/tmp.txt#thelinkofthefile")), FileInputFormat.addInputPath (job, new Path (args [0])), FileOutputFormat.setOutputPath (job, new Path (args [1])) when reading below. Boolean test = job.waitForCompletion (true); LOG.info ("End the main method."); System.exit (test?0:1);}} class GetFileNameMapper extends Mapper {private final Log LOG = LogFactory.getLog (GetFileNameMapper.class) @ Override protected void setup (Mapper.Context context) throws IOException, InterruptedException {if (context.getCacheFiles () .length > 0); URI u = context.getCacheFiles () [0]; / / use the link here to access the file BufferedReader br = new BufferedReader (new FileReader (new File (". / thelinkofthefile")) String line = br.readLine (); context.write (new Text (line), new Text ()); System.out.println ("Here I readLine:" + line) } @ Override protected void map (LongWritable key, Text value, Mapper.Context context) throws IOException, InterruptedException {}} Thank you for reading this article carefully. I hope the article "how hadoop avoids data skew through cachefile" shared by the editor will be helpful to you. At the same time, I also hope that you will support and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.