Shulou(Shulou.com)05/31 Report--

This article introduces the relevant knowledge of "how to realize the HelloWorld program in Hadoop". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

When you execute wordcount on the linux platform, there is an official example. The corresponding jar package is placed in hadoop-examples-2.0.0-mr1-cdh5.5.0.jar under hadoop-2.0.0-cdh5.5.0\ share\ hadoop\ mapreduce1 (Note: I am using the CDH4.5.0 version). We first need to prepare the data:

Echo "Hello World Hello Hadoop" > 1.txtecho "Hello Hadoop Bye" > 2.txt

Then put the data into HDFS:

Hadoop fs-mkdir / inputhadoop fs-put / root/1.txt / inputhadoop fs-put / root/2.txt / input

And then go to the directory where jar is located. "

Cd hadoop-2.0.0-cdh5.5.0\ share\ hadoop\ mapreduce1

Execute the command:

Hadoop jar hadoop-mapreduce-examples-2.0.0-cdh5.5.0.jar WordCount / input / output

Where / output is the output directory of the execution result.

At this point, HelloWorld is successfully executed, you can use the hadoop fs-cat / output/part command to see the results.

Next, we'll take a look at how eclipse is executed on window.

First post the code:

Public class WordCount {/ / mapper public static class Map extends Mapper {private static IntWritable one = new IntWritable (1); private Text word = new Text (); @ Override public void map (LongWritable key, Text value, Context context) throws IOException, InterruptedException {String line = value.toString (); StringTokenizer token = new StringTokenizer (line) While (token.hasMoreElements ()) {word.set (token.nextToken ()); context.write (word, one);}};} / reduce public static class Reduce extends Reducer {protected void reduce (Text key, Iterable values, Context context) throws IOException, InterruptedException {int sum = 0 For (IntWritable value: values) {sum + = value.get ();} context.write (key, new IntWritable (sum));} public static void main (String [] args) throws Exception {Configuration conf = new Configuration (); System.setProperty ("HADOOP_USER_NAME", "root") / / this sentence is important, otherwise you will be told that you do not have permission to execute Job job = new Job (conf); String [] ioArgs = new String [] {"hdfs://192.168.1.101:7001/input", "hdfs://192.168.1.101:7001/output"}; String [] otherArgs = new GenericOptionsParser (conf, ioArgs). GetRemainingArgs () Job.setJarByClass (WordCount.class); FileInputFormat.addInputPath (job, new Path (otherArgs [0])); FileOutputFormat.setOutputPath (job, new Path (otherArgs [1])); job.setMapperClass (Map.class); job.setReducerClass (Reduce.class); job.setOutputKeyClass (Text.class); job.setOutputValueClass (IntWritable.class); System.exit (job.waitForCompletion (true)? 0: 1);}}

Then click on the eclipse to execute. During execution, you may find that there is not enough memory in jvm. Just add the-Xmx1024M parameter to execute.

This is the end of the content of "how to implement HelloWorld programs in Hadoop". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.