Shulou(Shulou.com)06/02 Report--

This article focuses on "how to package and run MapReduce programs in Hadoop". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn "how Hadoop packages and runs MapReduce programs".

Main content: pack the MapReduce code into a jar package through the command line, and then submit it to the Hadoop cluster to run. The WordCount.java and WordCount.txt of the example are shown at the end.

First, the application that compiles Hadoop needs to add the required dependency package to CLASSPATH, which can be added to .bashrc or / etc/profile.

# javac compilation related packages rely on HADOOP_CLASSPATH=$ ($HADOOP_HOME/bin/hadoop classpath) # add HADOOP_CLASSPATH to CLASSPATHexport CLASSPATH=.:$HADOOP_CLASSPATH:$CLASSPATH

II. Compile the source code

# compilation does not set CLASSPATH to package jar-cvf WordCount.jar * .class via-cp $($HADOOP_HOME/bin/hadoop classpath) javac WordCount.java#

3. Submit to Hadoop

# upload WordCount.txt to Hadoophdfs dfs-mkdir inputhdfs dfs-put WordCount.txt input# submit task jar package, main class, input folder, output folder hadoop jar WordCount.jar WordCount input output# View run result hdfs dfs-cat output/*# delete output result directory hdfs dfs-rm-r output

IV. Running results

And 1bigdata 2hadoop 2hello 4world 1

Appendix:

WordCount.txt, words are separated by spaces

Hello worldhello hadoophello bigdatahello hadoop and bigdata

WordCount.java

Import java.io.IOException;import java.util.StringTokenizer;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.Reducer;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat Public class WordCount {public static class TokenizerMapper extends Mapper {private final static IntWritable ONE = new IntWritable (1); private final Text word = new Text (); @ Override public void map (Object key, Text value, Context context) throws IOException, InterruptedException {StringTokenizer itr = new StringTokenizer (value.toString ()); while (itr.hasMoreTokens ()) {word.set (itr.nextToken ()) Context.write (word, ONE);}} public static class IntSumReducer extends Reducer {private final IntWritable result = new IntWritable (); @ Override public void reduce (Text key, Iterable values, Context context) throws IOException, InterruptedException {int sum = 0; for (IntWritableval: values) {sum + = val.get () } result.set (sum); context.write (key, result);}} public static void main (String [] args) throws Exception {Configuration conf = new Configuration (); Job job = Job.getInstance (conf, "WordCount"); job.setJarByClass (WordCount.class); job.setMapperClass (TokenizerMapper.class); job.setCombinerClass (IntSumReducer.class) Job.setReducerClass (IntSumReducer.class); job.setOutputKeyClass (Text.class); job.setOutputValueClass (IntWritable.class); FileInputFormat.addInputPath (job, new Path (args [0])); FileOutputFormat.setOutputPath (job, new Path (args [1])); System.exit (job.waitForCompletion (true)? 0: 1) }} at this point, I believe you have a deeper understanding of "how Hadoop packages and runs MapReduce programs". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.