Shulou(Shulou.com)05/31 Report--

Editor to share with you how hadoop2.7.1 manually compiled wordcount, I hope you will learn something after reading this article, let's discuss it together!

Hadoop 2.7.1 manually compile wordcount

Successfully set up the hadoop running environment and run wordcount, which is actually achieved by running the compiled program in the example provided by hadoop with the given input and output path, but in the real development process, we need to write java code ourselves, compile and package in order to run on hadoop, so this time, we use the WordCount.java source code provided by hadoop to manually compile, package, and finally call and run under Linux. It lays the foundation for further hadoop program development.

1 We need to find WordCount.java:

The source code of hadoop can be downloaded from http://mirror.esocc.com/apache/hadoop/common/stable/, decompressed, and found under hadoop-2.7.1-src/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/dadoop/examples. It also contains a lot of sample code, which you can study when you are free.

2 the author user name is sr, create a new wordcount_01 folder under the sr directory, and then create two folders, src and classes, to store .java and compiled .class files, respectively. Put the found WordCount.java file into src, open WordCount.java, I am ashamed to change the package name to package com.sr.hadoop, a little embarrassed.

3 "cd ~ / wordcount_01" enter the wordcount_01 folder

4 "javac-classpath / home/sr/usr/hadoop/hadoop-2.7.1/share/hadoop/common/hadoop-common-2.7.1.jar:/home/sr/usr/hadoop/hadoop-2.7.1/share/hadoop/common/lib/commons-cli-1.2.jar:/home/sr/usr/hadoop/hadoop-2.7.1/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.1.jar-d. / classes/. / src/*.java"

This step is more critical, the role is to use the hadoop installation environment hadoop-common-2.7.1.jar, commons-cli-1.2.jar, hadoop-mapreduce-client-core-2.7.1.jar three library files, compile all the .java files in src (we can also write as WordCount.java). -classpath is followed by three absolute paths, which are the locations of the three library files on the author's computer, and the absolute paths are split;-d is followed by the compiled storage path and the source path. After execution, you will find that a three-level file directory is generated under classes, in which there are compiled .class files, and you will find that for each class in .java, there is a corresponding file generation.

5 "jar-cvf Srwordcount.jar-C. / classes/." Package the compiled class file into a Jar package, and the packaged Srwordcount will appear in the wordcount_01 directory. Only the packaged .class file can run under hadoop.

6 run Srwordcount: pay attention to the need to cd to your own hadoop environment, mine is "cd ~ / usr/hadoop/hadoop-2.7.1", please make sure that hadoop starts, input places the text files that need to be run, and there is no output folder, because hadoop will not write data to the existing folder in order to prevent the running results from being overwritten. If you have a problem with any of the above steps, please check my previous blog.

"bin/hadoop jar / home/sr/wordcount_01/Srwordcount.jar com.sr.hadoop.WordCount / input / output" means that the contents of input are run with packaged Srwordcount.jar and put into output. Note that com.sr.hadoop.WordCount invokes WordCount.class in the three-tier directory, and I changed the package name in step 2.

After reading this article, I believe you have a certain understanding of "how hadoop2.7.1 compiles wordcount manually". If you want to know more about it, you are welcome to follow the industry information channel. Thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.