Shulou(Shulou.com)05/31 Report--

This article mainly explains "the process of building hadoop". Friends who are interested may wish to have a look. The method introduced in this paper is simple, fast and practical. Now let the editor take you to learn "the process of building hadoop"!

! The rz command cannot be used: yum install lrzsz1. Go to http://hadoop.apache.org/ to download hadoop. 64-bit download hadoop-x.x.x-src.tar.gz 32-bit download hadoop-2.8.1.tar.gz***2.JAVA installation

[root@hadoop001 ~] # rz # upload jdk-8u45-linux-x64.gz

[root@hadoop001] # mkdir-p / usr/java

[root@hadoop001 ~] # mv jdk-8u45-linux-x64.gz / usr/java

[root@hadoop001 ~] # cd / usr/java

[root@hadoop001 ~] # tar-xzvf jdk-8u45-linux-x64.gz

# modify users and user groups

[root@hadoop002 java] # ll

Total 169388

Drwxr-xr-x. 8 uucp 143 4096 Apr 11 2015 jdk1.8.0_45

-rw-r--r--. 1 root root 173271626 Mar 16 15:25 jdk-8u45-linux-x64.gz

[root@hadoop002 java] # chown-R root:root jdk1.8.0_45

[root@hadoop002 java] # ll

Total 169388

Drwxr-xr-x. 8 root root 4096 Apr 11 2015 jdk1.8.0_45

-rw-r--r--. 1 root root 173271626 Mar 16 15:25 jdk-8u45-linux-x64.gz

[root@hadoop002 java] #

[root@hadoop001 java] # vi / etc/profile

Export JAVA_HOME=/usr/java/jdk1.8.0_45

Export PATH=$JAVA_HOME/bin:$PATH

[root@hadoop001 java] # source / etc/profile

[root@hadoop001 java] # which java

/ usr/java/jdk1.8.0_45/bin/java

[root@hadoop001 java] # java-version

Java version "1.8.045"

Java (TM) SE Runtime Environment (build 1.8.0_45-b14)

Java HotSpot (TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

[root@hadoop001 java] #

3.Maven installation

[root@hadoop001] # mkdir-p / opt/sourcecode / opt/software

[root@hadoop001 ~] # cd / opt/software/

[root@hadoop001 software] # rz

Rz waiting to receive.

Starting zmodem transfer. Press Ctrl+C to cancel.

Transferring apache-maven-3.3.9-bin.zip...

8415 KB 8415 KB/sec 00:00:01 0 Errors

[root@hadoop001 software] #

[root@hadoop001 software] # ll

Total 8432

-rw-r--r--. 1 root root 8617253 Aug 20 12:35 apache-maven-3.3.9-bin.zip

[root@hadoop001 software] # unzip apache-maven-3.3.9-bin.zip

[root@hadoop001 java] # vi / etc/profile

Export MAVEN_HOME=/opt/software/apache-maven-3.3.9

Export MAVEN_OPTS= "- Xms256m-Xmx512m"

Export PATH=$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH

[root@hadoop001 ~] # mvn-version

Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)

Maven home: / opt/software/apache-maven-3.3.9

Java version: 1.8.0_45, vendor: Oracle Corporation

Java home: / usr/java/jdk1.8.0_45/jre

Default locale: en_US, platform encoding: UTF-8

OS name: "linux", version: "2.6.32-431.el6.x86_64", arch: "amd64", family: "unix"

[root@hadoop001 ~] #

4.protobuf installation

[root@hadoop001 software] # rz

Rz waiting to receive.

Starting zmodem transfer. Press Ctrl+C to cancel.

Transferring protobuf-2.5.0.tar.gz...

2345 KB 2345 KB/sec 00:00:01 0 Errors

[root@hadoop001 software] # tar-xzvf protobuf-2.5.0.tar.gz

[root@hadoop001 software] # ll

Total 10792

Drwxr-xr-x. 6 root root 4096 Nov 10 2015 apache-maven-3.3.9

-rw-r--r--. 1 root root 8617253 Aug 20 12:35 apache-maven-3.3.9-bin.zip

Drwxr-xr-x. 10 109965 5000 4096 Feb 27 2013 protobuf-2.5.0

-rw-r--r--. 1 root root 2401901 Aug 20 13:03 protobuf-2.5.0.tar.gz

[root@hadoop001 software] # cd protobuf-2.5.0

[root@hadoop001 protobuf-2.5.0] # yum install-y gcc gcc-c++ make cmake # has installed cmake

[root@hadoop001 protobuf-2.5.0] #. / configure-prefix=/usr/local/protobuf

[root@hadoop001 protobuf-2.5.0] # make & & make install

[root@hadoop001 java] # vi / etc/profile

Export PROTOC_HOME=/usr/local/protobuf

Export PATH=$PROTOC_HOME/bin:$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH

[root@hadoop001 protobuf-2.5.0] # source / etc/profile

[root@hadoop001 protobuf-2.5.0] # protoc-- version

Libprotoc 2.5.0

[root@hadoop001 protobuf-2.5.0] #

5.Findbugs installation

[root@hadoop001 software] # rz

Rz waiting to receive.

Starting zmodem transfer. Press Ctrl+C to cancel.

Transferring findbugs-1.3.9.zip...

7369 KB 7369 KB/sec 00:00:01 0 Errors

[root@hadoop001 software] # unzip findbugs-1.3.9.zip

[root@hadoop002 software] # vi / etc/profile

Export FINDBUGS_HOME=/opt/software/findbugs-1.3.9

Export PATH=$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH

[root@hadoop001 software] #

[root@hadoop001 software] # source / etc/profile

[root@hadoop001 software] # findbugs-version

1.3.9

[root@hadoop001 software] #

6. Other dependencies

Yum install-y openssl openssl-devel svn ncurses-devel zlib-devel libtool

Yum install-y snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop autoconf automake

7. Compile

[root@hadoop001 sourcecode] # cd hadoop-2.8.1-src

[root@hadoop001 hadoop-2.8.1-src] # mvn clean package-Pdist,native-DskipTests-Dtar

[INFO]-maven-site-plugin:3.5:attach-descriptor (attach-descriptor) @ hadoop-dist-

[INFO]

[INFO] Reactor Summary:

[INFO]

[INFO] Apache Hadoop Main.. SUCCESS [7.770 s]

[INFO] Apache Hadoop Build Tools.. SUCCESS [4.990 s]

[INFO] Apache Hadoop Project POM.. SUCCESS [4.920 s]

[INFO] Apache Hadoop Annotations.. SUCCESS [12.936 s]

[INFO] Apache Hadoop Assemblies.. SUCCESS [0.809 s]

[INFO] Apache Hadoop Project Dist POM. SUCCESS [5.013 s]

[INFO] Apache Hadoop Maven Plugins.... SUCCESS [17.617 s]

[INFO] Apache Hadoop MiniKDC.. SUCCESS [23.551 s]

[INFO] Apache Hadoop Auth.. SUCCESS [26.352 s]

[INFO] Apache Hadoop Auth Examples.... SUCCESS [10.305 s]

[INFO] Apache Hadoop Common.. SUCCESS [05:58 min]

[INFO] Apache Hadoop NFS.. SUCCESS [19.105 s]

[INFO] Apache Hadoop KMS.. SUCCESS [27.479 s]

[INFO] Apache Hadoop Common Project... SUCCESS [0.174 s]

[INFO] Apache Hadoop HDFS Client.. SUCCESS [01:32 min]

[INFO] Apache Hadoop HDFS.. SUCCESS [06:42 min]

[INFO] Apache Hadoop HDFS Native Client. SUCCESS [20.370 s]

[INFO] Apache Hadoop HttpFS.. SUCCESS [40.720 s]

[INFO] Apache Hadoop HDFS BookKeeper Journal. SUCCESS [21.298 s]

[INFO] Apache Hadoop HDFS-NFS.. SUCCESS [11.818 s]

[INFO] Apache Hadoop HDFS Project. SUCCESS [0.148 s]

[INFO] Apache Hadoop YARN.. SUCCESS [0.192 s]

[INFO] Apache Hadoop YARN API.. SUCCESS [41.517 s]

[INFO] Apache Hadoop YARN Common.. SUCCESS [01:19 min]

[INFO] Apache Hadoop YARN Server.. SUCCESS [0.192 s]

[INFO] Apache Hadoop YARN Server Common. SUCCESS [19.421 s]

[INFO] Apache Hadoop YARN NodeManager. SUCCESS [42.398 s]

[INFO] Apache Hadoop YARN Web Proxy... SUCCESS [8.925 s]

[INFO] Apache Hadoop YARN ApplicationHistoryService. SUCCESS [16.120 s]

[INFO] Apache Hadoop YARN ResourceManager. SUCCESS [57.415 s]

[INFO] Apache Hadoop YARN Server Tests. SUCCESS [3.869 s]

[INFO] Apache Hadoop YARN Client.. SUCCESS [14.325 s]

[INFO] Apache Hadoop YARN SharedCacheManager. SUCCESS [11.814 s]

[INFO] Apache Hadoop YARN Timeline Plugin Storage. SUCCESS [10.027 s]

[INFO] Apache Hadoop YARN Applications. SUCCESS [0.276 s]

[INFO] Apache Hadoop YARN DistributedShell. SUCCESS [8.333 s]

[INFO] Apache Hadoop YARN Unmanaged Am Launcher. SUCCESS [5.473 s]

[INFO] Apache Hadoop YARN Site.. SUCCESS [0.160 s]

[INFO] Apache Hadoop YARN Registry.... SUCCESS [13.204 s]

[INFO] Apache Hadoop YARN Project. SUCCESS [8.106 s]

[INFO] Apache Hadoop MapReduce Client. SUCCESS [0.514 s]

[INFO] Apache Hadoop MapReduce Core... SUCCESS [01:09 min]

[INFO] Apache Hadoop MapReduce Common. SUCCESS [40.479 s]

[INFO] Apache Hadoop MapReduce Shuffle. SUCCESS [10.304 s]

[INFO] Apache Hadoop MapReduce App.... SUCCESS [27.335 s]

[INFO] Apache Hadoop MapReduce HistoryServer. SUCCESS [19.910 s]

[INFO] Apache Hadoop MapReduce JobClient. SUCCESS [16.657 s]

[INFO] Apache Hadoop MapReduce HistoryServer Plugins. SUCCESS [4.591 s]

[INFO] Apache Hadoop MapReduce Examples. SUCCESS [12.346 s]

[INFO] Apache Hadoop MapReduce.. SUCCESS [5.966 s]

[INFO] Apache Hadoop MapReduce Streaming. SUCCESS [7.940 s]

[INFO] Apache Hadoop Distributed Copy. SUCCESS [15.245 s]

[INFO] Apache Hadoop Archives.. SUCCESS [5.380 s]

[INFO] Apache Hadoop Archive Logs. SUCCESS [5.812 s]

[INFO] Apache Hadoop Rumen.. SUCCESS [11.785 s]

[INFO] Apache Hadoop Gridmix.. SUCCESS [9.890 s]

[INFO] Apache Hadoop Data Join.. SUCCESS [5.784 s]

[INFO] Apache Hadoop Ant Tasks.. SUCCESS [3.254 s]

[INFO] Apache Hadoop Extras.. SUCCESS [5.495 s]

[INFO] Apache Hadoop Pipes.. SUCCESS [10.630 s]

[INFO] Apache Hadoop OpenStack support. SUCCESS [11.234 s]

[INFO] Apache Hadoop Amazon Web Services support. SUCCESS [14.060 s]

[INFO] Apache Hadoop Azure support.... SUCCESS [10.535 s]

[INFO] Apache Hadoop Client.. SUCCESS [13.519 s]

[INFO] Apache Hadoop Mini-Cluster. SUCCESS [2.164 s]

[INFO] Apache Hadoop Scheduler Load Simulator. SUCCESS [10.405 s]

[INFO] Apache Hadoop Tools Dist.. SUCCESS [11.514 s]

[INFO] Apache Hadoop Azure Data Lake support. SUCCESS [9.201 s]

[INFO] Apache Hadoop Tools.. SUCCESS [0.129 s]

[INFO] Apache Hadoop Distribution. SUCCESS [01:07 min]

[INFO]

[INFO] BUILD SUCCESS

[INFO]

[INFO] Total time: 31:41 min

[INFO] Finished at: 2017-12-10T11:55:28+08:00

[INFO] Final Memory: 166M/494M

[INFO]

[root@rzdatahadoop001 hadoop-2.8.1-src] #

It takes 3.5 hours for download

If you don't want to download, use .m2.tar.gz-- > extract-- > root home directory

8. Compiled tar package

/ opt/sourcecode/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz

Reminder:

1. Sometimes it takes too long to download a package during compilation, which is due to fake death in the process of connecting to the website.

Press ctrl+c at this time to rerun the compile command.

2. If a file is missing, clean up maven (use the command mvn clean) before recompiling.

At this point, I believe you have a deeper understanding of the "hadoop building process", might as well come to the actual operation of it! Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.