Shulou(Shulou.com)06/02 Report--

This article is to share with you the content of the compilation example of Hadoop under CentOS6.5. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

Hadoop introduction

(1) hadoop.apache.org

(2) narrow sense: software (HDFS, MapReduce, Yarn)

Commendatory meaning: biosphere dominated by hadoop

(3) Hadoop1.x

HDFS: distributed file system, Stora

MapReduce: distributed computing + resource scheduling and job scheduling

(4) Hadoop2.x

HDFS: Stora

MapReduce: calculation

Yarn: YetAnother Resource Negotiator resource scheduling and job scheduling platform

1. Check to see if there is a running hadoop.

Ps-ef | grep hadoop find /-name hadoop

2 create two sourcecode software folders under the / opt directory.

3. Upload the tar package of hadoop to sourcecode.

4. Extract the hadoop-2.8.1-src.tar.gz.

Tar-xzvf hadoop-2.8.1-src.tar.gz

5. View the environment compiled by Hadoop

Cd hadoop-2.8.1-src > > cat BUILDING.txt

6. Install JDK

7. Configure the JDK environment variable in / etc/profile

8 upload maven package

9. Configure the maven environment variable-/ etc/profile

10. Upload and install protobuf-2.5.0.tar

11Findbugs installation

[root@hadoop01 software] # rz

Rz waiting to receive.

Starting zmodem transfer. Press Ctrl+C to cancel.

Transferring findbugs-1.3.9.zip...

7369 KB 7369 KB/sec 00:00:01 0 Errors

[root@hadoop01 software] # unzip findbugs-1.3.9.zip

[root@hadoop01 software] # vi / etc/profile

Export FINDBUGS_HOME=/opt/software/findbugs-1.3.9

Export PATH=$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH

[root@hadoop01 software] #

[root@hadoop01 software] # source / etc/profile

[root@hadoop01 software] # findbugs-version

1.3.9

[root@hadoop001 software] #

twelve。 Compile

[root@hadoop01 sourcecode] # cd hadoop-2.8.1-src

[root@hadoop01 hadoop-2.8.1-src] # mvn clean package-Pdist,native-DskipTests-Dtar

[INFO]-maven-site-plugin:3.5:attach-descriptor (attach-descriptor) @ hadoop-dist-

[INFO]

[INFO] Reactor Summary:

[INFO]

[INFO] Apache Hadoop Main.. SUCCESS [7.770 s]

[INFO] Apache Hadoop Build Tools.. SUCCESS [4.990 s]

[INFO] Apache Hadoop Project POM.. SUCCESS [4.920 s]

[INFO] Apache Hadoop Annotations.. SUCCESS [12.936 s]

[INFO] Apache Hadoop Assemblies.. SUCCESS [0.809 s]

[INFO] Apache Hadoop Project Dist POM. SUCCESS [5.013 s]

[INFO] Apache Hadoop Maven Plugins.... SUCCESS [17.617 s]

[INFO] Apache Hadoop MiniKDC.. SUCCESS [23.551 s]

[INFO] Apache Hadoop Auth.. SUCCESS [26.352 s]

[INFO] Apache Hadoop Auth Examples.... SUCCESS [10.305 s]

[INFO] Apache Hadoop Common.. SUCCESS [05:58 min]

[INFO] Apache Hadoop NFS.. SUCCESS [19.105 s]

[INFO] Apache Hadoop KMS.. SUCCESS [27.479 s]

[INFO] Apache Hadoop Common Project... SUCCESS [0.174 s]

[INFO] Apache Hadoop HDFS Client.. SUCCESS [01:32 min]

[INFO] Apache Hadoop HDFS.. SUCCESS [06:42 min]

[INFO] Apache Hadoop HDFS Native Client. SUCCESS [20.370 s]

[INFO] Apache Hadoop HttpFS.. SUCCESS [40.720 s]

[INFO] Apache Hadoop HDFS BookKeeper Journal. SUCCESS [21.298 s]

[INFO] Apache Hadoop HDFS-NFS.. SUCCESS [11.818 s]

[INFO] Apache Hadoop HDFS Project. SUCCESS [0.148 s]

[INFO] Apache Hadoop YARN.. SUCCESS [0.192 s]

[INFO] Apache Hadoop YARN API.. SUCCESS [41.517 s]

[INFO] Apache Hadoop YARN Common.. SUCCESS [01:19 min]

[INFO] Apache Hadoop YARN Server.. SUCCESS [0.192 s]

[INFO] Apache Hadoop YARN Server Common. SUCCESS [19.421 s]

[INFO] Apache Hadoop YARN NodeManager. SUCCESS [42.398 s]

[INFO] Apache Hadoop YARN Web Proxy... SUCCESS [8.925 s]

[INFO] Apache Hadoop YARN ApplicationHistoryService. SUCCESS [16.120 s]

[INFO] Apache Hadoop YARN ResourceManager. SUCCESS [57.415 s]

[INFO] Apache Hadoop YARN Server Tests. SUCCESS [3.869 s]

[INFO] Apache Hadoop YARN Client.. SUCCESS [14.325 s]

[INFO] Apache Hadoop YARN SharedCacheManager. SUCCESS [11.814 s]

[INFO] Apache Hadoop YARN Timeline Plugin Storage. SUCCESS [10.027 s]

[INFO] Apache Hadoop YARN Applications. SUCCESS [0.276 s]

[INFO] Apache Hadoop YARN DistributedShell. SUCCESS [8.333 s]

[INFO] Apache Hadoop YARN Unmanaged Am Launcher. SUCCESS [5.473 s]

[INFO] Apache Hadoop YARN Site.. SUCCESS [0.160 s]

[INFO] Apache Hadoop YARN Registry.... SUCCESS [13.204 s]

[INFO] Apache Hadoop YARN Project. SUCCESS [8.106 s]

[INFO] Apache Hadoop MapReduce Client. SUCCESS [0.514 s]

[INFO] Apache Hadoop MapReduce Core... SUCCESS [01:09 min]

[INFO] Apache Hadoop MapReduce Common. SUCCESS [40.479 s]

[INFO] Apache Hadoop MapReduce Shuffle. SUCCESS [10.304 s]

[INFO] Apache Hadoop MapReduce App.... SUCCESS [27.335 s]

[INFO] Apache Hadoop MapReduce HistoryServer. SUCCESS [19.910 s]

[INFO] Apache Hadoop MapReduce JobClient. SUCCESS [16.657 s]

[INFO] Apache Hadoop MapReduce HistoryServer Plugins. SUCCESS [4.591 s]

[INFO] Apache Hadoop MapReduce Examples. SUCCESS [12.346 s]

[INFO] Apache Hadoop MapReduce.. SUCCESS [5.966 s]

[INFO] Apache Hadoop MapReduce Streaming. SUCCESS [7.940 s]

[INFO] Apache Hadoop Distributed Copy. SUCCESS [15.245 s]

[INFO] Apache Hadoop Archives.. SUCCESS [5.380 s]

[INFO] Apache Hadoop Archive Logs. SUCCESS [5.812 s]

[INFO] Apache Hadoop Rumen.. SUCCESS [11.785 s]

[INFO] Apache Hadoop Gridmix.. SUCCESS [9.890 s]

[INFO] Apache Hadoop Data Join.. SUCCESS [5.784 s]

[INFO] Apache Hadoop Ant Tasks.. SUCCESS [3.254 s]

[INFO] Apache Hadoop Extras.. SUCCESS [5.495 s]

[INFO] Apache Hadoop Pipes.. SUCCESS [10.630 s]

[INFO] Apache Hadoop OpenStack support. SUCCESS [11.234 s]

[INFO] Apache Hadoop Amazon Web Services support. SUCCESS [14.060 s]

[INFO] Apache Hadoop Azure support.... SUCCESS [10.535 s]

[INFO] Apache Hadoop Client.. SUCCESS [13.519 s]

[INFO] Apache Hadoop Mini-Cluster. SUCCESS [2.164 s]

[INFO] Apache Hadoop Scheduler Load Simulator. SUCCESS [10.405 s]

[INFO] Apache Hadoop Tools Dist.. SUCCESS [11.514 s]

[INFO] Apache Hadoop Azure Data Lake support. SUCCESS [9.201 s]

[INFO] Apache Hadoop Tools.. SUCCESS [0.129 s]

[INFO] Apache Hadoop Distribution. SUCCESS [01:07 min]

[INFO]

[INFO] BUILD SUCCESS

[INFO]

[INFO] Total time: 31:41 min

[INFO] Finished at: 2017-12-10T11:55:28+08:00

[INFO] Final Memory: 166M/494M

[INFO]

[root@rzdatahadoop01 hadoop-2.8.1-src] #

Note: it takes 3.5 hours for download

If you don't want to download, use .m2.tar.gz-- > extract-- > root home directory

13 compiled tar package

/ opt/sourcecode/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz

Reminder:

1. Sometimes it takes too long to download a package during compilation, which is due to fake death in the process of connecting to the website.

Press ctrl+c at this time to rerun the compile command.

2. If a file is missing, clean up maven (use the command mvn clean) before recompiling.

Thank you for reading! This is the end of this article on "compiling examples of Hadoop under CentOS6.5". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it out for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.