How to compile hadoop-2.8.1

This article mainly shows you "how to compile hadoop-2.8.1", the content is easy to understand, clear, hope to help you solve your doubts, the following let the editor lead you to study and learn "how to compile hadoop-2.8.1" this article.

1. Download the hadoop-2.8.1 source code

# first create software and sourcecode folders under / opt

[root@hadoop001] # mkdir-p / opt/sourcecode / opt/software

[root@hadoop001 sourcecode] # cd / opt/sourcecode

[root@hadoop001 sourcecode] # pwd

/ opt/sourcecode

[root@hadoop001 sourcecode] # ll

Total 33756

-rw-r--r--. 1 root root 34523353 Aug 20 12:14 hadoop-2.8.1-src.tar.gz

two。 Extract the hadoop-2.8.1-src.tar.gz file

[root@hadoop001 sourcecode] # tar-xzvf hadoop-2.8.1-src.tar.gz

[root@hadoop001 sourcecode] # ll

Total 33760

Drwxr-xr-x. 17 root root 4096 Jun 2 14:13 hadoop-2.8.1-src

-rw-r--r--. 1 root root 34523353 Aug 20 12:14 hadoop-2.8.1-src.tar.gz

3. Open the hadoop-2.8.1-sr/BUILDING.txt file to see the other components needed to compile hadoop-2.8.1

[root@hadoop001 sourcecode] # cd hadoop-2.8.1-src

[root@hadoop001 hadoop-2.8.1-src] # ll

Total 224

-rw-rw-r--. 1 root root 15623 May 24 07:14 BUILDING.txt

Drwxr-xr-x. 4 root root 4096 Aug 20 12:44 dev-support

Drwxr-xr-x. 3 root root 4096 Aug 20 12:44 hadoop-assemblies

Drwxr-xr-x. 3 root root 4096 Aug 20 12:44 hadoop-build-tools

Drwxrwxr-x. 2 root root 4096 Aug 20 12:44 hadoop-client

Drwxr-xr-x. 10 root root 4096 Aug 20 12:44 hadoop-common-project

Drwxr-xr-x. 2 root root 4096 Aug 20 12:44 hadoop-dist

Drwxr-xr-x. 8 root root 4096 Aug 20 12:44 hadoop-hdfs-project

Drwxr-xr-x. 9 root root 4096 Aug 20 12:44 hadoop-mapreduce-project

Drwxr-xr-x. 3 root root 4096 Aug 20 12:44 hadoop-maven-plugins

Drwxr-xr-x. 2 root root 4096 Aug 20 12:44 hadoop-minicluster

Drwxr-xr-x. 3 root root 4096 Aug 20 12:44 hadoop-project

Drwxr-xr-x. 2 root root 4096 Aug 20 12:44 hadoop-project-dist

Drwxr-xr-x. 18 root root 4096 Aug 20 12:44 hadoop-tools

Drwxr-xr-x. 3 root root 4096 Aug 20 12:44 hadoop-yarn-project

-rw-rw-r--. 1 root root 99253 May 24 07:14 LICENSE.txt

-rw-rw-r--. 1 root root 15915 May 24 07:14 NOTICE.txt

Drwxrwxr-x. 2 root root 4096 Jun 2 14:24 patchprocess

-rw-rw-r--. 1 root root 20477 May 29 06:36 pom.xml

-rw-r--r--. 1 root root 1366 May 20 13:30 README.txt

-rwxrwxr-x. 1 root root 1841 May 24 07:14 start-build-env.sh

[root@hadoop001 hadoop-2.8.1-src] #

[root@hadoop001 hadoop-2.8.1-src] # cat BUILDING.txt

Build instructions for Hadoop

Requirements:

* Unix System

* JDK 1.7 +

* Maven 3.0 or later

* Findbugs 1.3.9 (if running findbugs)

* ProtocolBuffer 2.5.0

* CMake 2.6or newer (if compiling native code), must be 3.0or newer on Mac

* Zlib devel (if compiling native code)

* openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)

* Linux FUSE (Filesystem in Userspace) version 2.6or above (if compiling fuse_dfs)

* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

4.JAVA installation must be JDK 1.7 +

# if you do not install rz, you need to install rz first

[root@hadoop001 ~] # yum install-y lrzsz

[root@hadoop001 ~] # rz # upload jdk-8u45-linux-x64.gz

[root@hadoop001] # mkdir-p / usr/java

[root@hadoop001 ~] # mv jdk-8u45-linux-x64.gz / usr/java

[root@hadoop001 ~] # cd / usr/java

[root@hadoop001 ~] # tar-xzvf jdk-8u45-linux-x64.gz

# modify users and user groups

[root@hadoop001 java] # ll

Total 169388

Drwxr-xr-x. 8 uucp 143 4096 Apr 11 2015 jdk1.8.0_45

-rw-r--r--. 1 root root 173271626 Mar 16 15:25 jdk-8u45-linux-x64.gz

[root@hadoop001 java] # chown-R root:root jdk1.8.0_45

[root@hadoop001 java] # ll

Total 169388

Drwxr-xr-x. 8 root root 4096 Apr 11 2015 jdk1.8.0_45

-rw-r--r--. 1 root root 173271626 Mar 16 15:25 jdk-8u45-linux-x64.gz

[root@hadoop001 java] #

# configure environment variables

[root@hadoop001 java] # vi / etc/profile

Export JAVA_HOME=/usr/java/jdk1.8.0_45

Export PATH=$JAVA_HOME/bin:$PATH

# make the environment variable effective

[root@hadoop001 java] # source / etc/profile

[root@hadoop001 java] # which java

/ usr/java/jdk1.8.0_45/bin/java

[root@hadoop001 java] # java-version

Java version "1.8.045"

Java (TM) SE Runtime Environment (build 1.8.0_45-b14)

Java HotSpot (TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

[root@hadoop001 java] #

# if the system still displays the default jdk version, you need to change the default jdk version to the currently installed version

Update-alternatives-install / usr/bin/java java/ usr/java/jdk1.8.0_45/bin/java 300

Update-alternatives-install / usr/bin/javac javac / usr/java/jdk1.8.0_45/bin/javac 300

Update-alternatives-config java

Update-alternatives-config javac

5.Maven installs Maven 3.0 or later

[root@hadoop001 ~] # cd / opt/software/

# upload apache-maven-3.3.9-bin.zip

[root@hadoop001 software] # rz

Rz waiting to receive.

Starting zmodem transfer. Press Ctrl+C to cancel.

Transferring apache-maven-3.3.9-bin.zip...

8415 KB 8415 KB/sec 00:00:01 0 Errors

[root@hadoop001 software] #

[root@hadoop001 software] # ll

Total 8432

-rw-r--r--. 1 root root 8617253 Aug 20 12:35 apache-maven-3.3.9-bin.zip

# decompress apache-maven-3.3.9-bin.zip

[root@hadoop001 software] # unzip apache-maven-3.3.9-bin.zip

# configure the environment variables of apache-maven-3.3.9

[root@hadoop001 software] # vi / etc/profile

Export MAVEN_HOME=/opt/software/apache-maven-3.3.9

Export MAVEN_OPTS= "- Xms256m-Xmx512m"

Export PATH=$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH

# make the environment variable effective

[root@hadoop001 software] # source / etc/profile

# View maven version

[root@hadoop001 software] # mvn-version

Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)

Maven home: / opt/software/apache-maven-3.3.9

Java version: 1.8.0_45, vendor: Oracle Corporation

Java home: / usr/java/jdk1.8.0_45/jre

Default locale: en_US, platform encoding: UTF-8

OS name: "linux", version: "2.6.32-431.el6.x86_64", arch: "amd64", family: "unix"

[root@hadoop001 software] #

6.Findbugs installs Findbugs 1.3.9 (if running findbugs)

# upload findbugs-1.3.9.zip

[root@hadoop001 software] # rz

Rz waiting to receive.

Starting zmodem transfer. Press Ctrl+C to cancel.

Transferring findbugs-1.3.9.zip...

7369 KB 7369 KB/sec 00:00:01 0 Errors

# decompress findbugs-1.3.9.zip

[root@hadoop001 software] # unzip findbugs-1.3.9.zip

# configure environment variables

[root@hadoop001 software] # vi / etc/profile

Export FINDBUGS_HOME=/opt/software/findbugs-1.3.9

Export PATH=$FINDBUGS_HOME/bin:$PROTOC_HOME/bin:$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH

# make the environment variable effective

[root@hadoop001 software] # source / etc/profile

# View Findbugs version

[root@hadoop001 software] # findbugs-version

1.3.9

[root@hadoop001 software] #

7.protobuf installs ProtocolBuffer 2.5.0

# upload protobuf-2.5.0.tar.gz

[root@hadoop001 software] # rz

Rz waiting to receive.

Starting zmodem transfer. Press Ctrl+C to cancel.

Transferring protobuf-2.5.0.tar.gz...

2345 KB 2345 KB/sec 00:00:01 0 Errors

# decompress protobuf-2.5.0.tar.gz

[root@hadoop001 software] # tar-xzvf protobuf-2.5.0.tar.gz

[root@hadoop001 software] # ll

Total 10792

Drwxr-xr-x. 6 root root 4096 Nov 10 2015 apache-maven-3.3.9

-rw-r--r--. 1 root root 8617253 Aug 20 12:35 apache-maven-3.3.9-bin.zip

Drwxr-xr-x. 10 109965 5000 4096 Feb 27 2013 protobuf-2.5.0

-rw-r--r--. 1 root root 2401901 Aug 20 13:03 protobuf-2.5.0.tar.gz

[root@hadoop001 software] # cd protobuf-2.5.0

[root@hadoop001 protobuf-2.5.0] # yum install-y gcc gcc-c++ make cmake # has installed cmake

[root@hadoop001 protobuf-2.5.0] #. / configure-prefix=/usr/local/protobuf

[root@hadoop001 protobuf-2.5.0] # make & & make install

# configure environment variables

[root@hadoop001 protobuf-2.5.0] # vi / etc/profile

Export PROTOC_HOME=/usr/local/protobuf

Export PATH=$PROTOC_HOME/bin:$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH

# make the environment variable effective

[root@hadoop001 protobuf-2.5.0] # source / etc/profile

# View protobuf version

[root@hadoop001 protobuf-2.5.0] # protoc-- version

Libprotoc 2.5.0

[root@hadoop001 protobuf-2.5.0] #

8. Other dependencies

Yum install-y openssl openssl-devel svn ncurses-devel zlib-devel libtool

Yum install-y snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop autoconf automake

9. Compile

[root@hadoop001 sourcecode] # cd hadoop-2.8.1-src

[root@hadoop001 hadoop-2.8.1-src] # mvn clean package-Pdist,native-DskipTests-Dtar

# Screenshot of successful compilation

[INFO]-maven-site-plugin:3.5:attach-descriptor (attach-descriptor) @ hadoop-dist-

[INFO]

[INFO] Reactor Summary:

[INFO]

[INFO] Apache Hadoop Main.. SUCCESS [7.770 s]

[INFO] Apache Hadoop Build Tools.. SUCCESS [4.990 s]

[INFO] Apache Hadoop Project POM.. SUCCESS [4.920 s]

[INFO] Apache Hadoop Annotations.. SUCCESS [12.936 s]

[INFO] Apache Hadoop Assemblies.. SUCCESS [0.809 s]

[INFO] Apache Hadoop Project Dist POM. SUCCESS [5.013 s]

[INFO] Apache Hadoop Maven Plugins.... SUCCESS [17.617 s]

[INFO] Apache Hadoop MiniKDC.. SUCCESS [23.551 s]

[INFO] Apache Hadoop Auth.. SUCCESS [26.352 s]

[INFO] Apache Hadoop Auth Examples.... SUCCESS [10.305 s]

[INFO] Apache Hadoop Common.. SUCCESS [05:58 min]

[INFO] Apache Hadoop NFS.. SUCCESS [19.105 s]

[INFO] Apache Hadoop KMS.. SUCCESS [27.479 s]

[INFO] Apache Hadoop Common Project... SUCCESS [0.174 s]

[INFO] Apache Hadoop HDFS Client.. SUCCESS [01:32 min]

[INFO] Apache Hadoop HDFS.. SUCCESS [06:42 min]

[INFO] Apache Hadoop HDFS Native Client. SUCCESS [20.370 s]

[INFO] Apache Hadoop HttpFS.. SUCCESS [40.720 s]

[INFO] Apache Hadoop HDFS BookKeeper Journal. SUCCESS [21.298 s]

[INFO] Apache Hadoop HDFS-NFS.. SUCCESS [11.818 s]

[INFO] Apache Hadoop HDFS Project. SUCCESS [0.148 s]

[INFO] Apache Hadoop YARN.. SUCCESS [0.192 s]

[INFO] Apache Hadoop YARN API.. SUCCESS [41.517 s]

[INFO] Apache Hadoop YARN Common.. SUCCESS [01:19 min]

[INFO] Apache Hadoop YARN Server.. SUCCESS [0.192 s]

[INFO] Apache Hadoop YARN Server Common. SUCCESS [19.421 s]

[INFO] Apache Hadoop YARN NodeManager. SUCCESS [42.398 s]

[INFO] Apache Hadoop YARN Web Proxy... SUCCESS [8.925 s]

[INFO] Apache Hadoop YARN ApplicationHistoryService. SUCCESS [16.120 s]

[INFO] Apache Hadoop YARN ResourceManager. SUCCESS [57.415 s]

[INFO] Apache Hadoop YARN Server Tests. SUCCESS [3.869 s]

[INFO] Apache Hadoop YARN Client.. SUCCESS [14.325 s]

[INFO] Apache Hadoop YARN SharedCacheManager. SUCCESS [11.814 s]

[INFO] Apache Hadoop YARN Timeline Plugin Storage. SUCCESS [10.027 s]

[INFO] Apache Hadoop YARN Applications. SUCCESS [0.276 s]

[INFO] Apache Hadoop YARN DistributedShell. SUCCESS [8.333 s]

[INFO] Apache Hadoop YARN Unmanaged Am Launcher. SUCCESS [5.473 s]

[INFO] Apache Hadoop YARN Site.. SUCCESS [0.160 s]

[INFO] Apache Hadoop YARN Registry.... SUCCESS [13.204 s]

[INFO] Apache Hadoop YARN Project. SUCCESS [8.106 s]

[INFO] Apache Hadoop MapReduce Client. SUCCESS [0.514 s]

[INFO] Apache Hadoop MapReduce Core... SUCCESS [01:09 min]

[INFO] Apache Hadoop MapReduce Common. SUCCESS [40.479 s]

[INFO] Apache Hadoop MapReduce Shuffle. SUCCESS [10.304 s]

[INFO] Apache Hadoop MapReduce App.... SUCCESS [27.335 s]

[INFO] Apache Hadoop MapReduce HistoryServer. SUCCESS [19.910 s]

[INFO] Apache Hadoop MapReduce JobClient. SUCCESS [16.657 s]

[INFO] Apache Hadoop MapReduce HistoryServer Plugins. SUCCESS [4.591 s]

[INFO] Apache Hadoop MapReduce Examples. SUCCESS [12.346 s]

[INFO] Apache Hadoop MapReduce.. SUCCESS [5.966 s]

[INFO] Apache Hadoop MapReduce Streaming. SUCCESS [7.940 s]

[INFO] Apache Hadoop Distributed Copy. SUCCESS [15.245 s]

[INFO] Apache Hadoop Archives.. SUCCESS [5.380 s]

[INFO] Apache Hadoop Archive Logs. SUCCESS [5.812 s]

[INFO] Apache Hadoop Rumen.. SUCCESS [11.785 s]

[INFO] Apache Hadoop Gridmix.. SUCCESS [9.890 s]

[INFO] Apache Hadoop Data Join.. SUCCESS [5.784 s]

[INFO] Apache Hadoop Ant Tasks.. SUCCESS [3.254 s]

[INFO] Apache Hadoop Extras.. SUCCESS [5.495 s]

[INFO] Apache Hadoop Pipes.. SUCCESS [10.630 s]

[INFO] Apache Hadoop OpenStack support. SUCCESS [11.234 s]

[INFO] Apache Hadoop Amazon Web Services support. SUCCESS [14.060 s]

[INFO] Apache Hadoop Azure support.... SUCCESS [10.535 s]

[INFO] Apache Hadoop Client.. SUCCESS [13.519 s]

[INFO] Apache Hadoop Mini-Cluster. SUCCESS [2.164 s]

[INFO] Apache Hadoop Scheduler Load Simulator. SUCCESS [10.405 s]

[INFO] Apache Hadoop Tools Dist.. SUCCESS [11.514 s]

[INFO] Apache Hadoop Azure Data Lake support. SUCCESS [9.201 s]

[INFO] Apache Hadoop Tools.. SUCCESS [0.129 s]

[INFO] Apache Hadoop Distribution. SUCCESS [01:07 min]

[INFO]

[INFO] BUILD SUCCESS

[INFO]

[INFO] Total time: 31:41 min

[INFO] Finished at: 2017-12-10T11:55:28+08:00

[INFO] Final Memory: 166M/494M

[INFO]

[root@hadoop001 hadoop-2.8.1-src] #

10. Compiled tar package

/ opt/sourcecode/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz

Reminder:

1. Sometimes it takes too long to download a package during compilation, which is due to fake death in the process of connecting to the website.

Press ctrl+c at this time to rerun the compile command.

2. If a file is missing, clean up maven (use the command mvn clean) before recompiling.

The above is all the content of this article "how to compile hadoop-2.8.1". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!