Shulou(Shulou.com)06/01 Report--

This article introduces the relevant knowledge of "MapReduce method of obtaining command line parameters in Hadoop". In the operation process of actual cases, many people will encounter such difficulties. Next, let Xiaobian lead you to learn how to handle these situations! I hope you can read carefully and learn something!

package cmd;import java.io.IOException;import java.net.URI;import java.net.URISyntaxException;import mapreduce.MyMapper;import mapreduce.MyReducer;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.conf.Configured;import org.apache.hadoop.fs.FileSystem;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.LongWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;import org.apache.hadoop.mapreduce.lib.partition.HashPartitioner;import org.apache.hadoop.util.Tool;import org.apache.hadoop.util.ToolRunner;/** * Compute words * @author Xr * */public class WordCountApp extends Configured implements Tool{ public static String INPUT_PATH = ""; public static String OUTPUT_PATH = ""; @Override public int run(String[] args) throws Exception { INPUT_PATH = args[0]; OUTPUT_PATH = args[1]; Configuration conf = new Configuration(); //Judge whether there is an input directory existsFile(conf); Job job = new Job(conf,WordCountApp.class.getName()); //make jar package job.setJarByClass(WordCountApp.class); //1.1 Where to read data FileInputFormat.setInputPaths(job, INPUT_PATH); //parse each line of input text into key-value pairs job.setInputFormatClass(TextInputFormat.class); //1.2 Set custom map function job.setMapperClass(MyMapper.class); job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(LongWritable.class); //1.3 partition job.setPartitionerClass(HashPartitioner.class); job.setNumReduceTasks(1); //1.4 TODO ordered group //1.5 TODO statute //2.1 It is done by the framework and does not require manual intervention by programmers. //2.2 Custom reducer function job.setReducerClass(MyReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(LongWritable.class); //2.3 Write to HDFS FileOutputFormat.setOutputPath(job, new Path(OUTPUT_PATH)); //formatting class job.setOutputFormatClass(TextOutputFormat.class); //submit to JobTracker for execution job.waitForCompletion(true); return 0; } public static void main(String[] args)throws Exception { ToolRunner.run(new WordCountApp(), args); } private static void existsFile(Configuration conf) throws IOException, URISyntaxException { FileSystem fs = FileSystem.get(new URI(INPUT_PATH), conf); if(fs.exists(new Path(OUTPUT_PATH))){ fs.delete(new Path(OUTPUT_PATH), true); } Hadoop WordCount.jar hdfs://hadoop:9000/hello hdfs://hadoop:9000/h2 Name : Xr Date : 2014-03-02 21:47"MapReduce in Hadoop to get command-line parameters" content is introduced here, thank you for reading. If you want to know more about industry-related knowledge, you can pay attention to the website. Xiaobian will output more high-quality practical articles for everyone!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.