Shulou(Shulou.com)06/03 Report--

Software version:

Jdk:1.8

Maven:3.61 http://maven.apache.org/download.cgi

Spark:2.42 https://archive.apache.org/dist/spark/spark-2.4.2/

Hadoop version: hadoop-2.6.0-cdh6.7.0 (hadoop version supported by spark compilation, no installation required)

Configure maven:

# configure environment variable [root@hadoop004 soft] # cat / etc/profile.d/maven.sh MAVEN_HOME=/usr/local/mavenexport PATH=$MAVEN_HOME/bin:$PATH# confirm maven version [root@hadoop004 maven] # mvn-- versionApache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555 2019-04-05T03:00:29+08:00) Maven home: / usr/local/mavenJava version: 1.8.0U111, vendor: Oracle Corporation, runtime: / usr/java/jdk1.8.0_111/jreDefault locale: en_US, platform encoding: UTF-8OS name: "linux", version: "3.10.0-862.3.2.el7.x86_64", arch: "amd64" Family: "unix" # configure the local storage address of mvn: settings.xml file / usr/local/maven/repo# configuration mvn download source is Aliyun's maven repository Accelerate the download of alimaven aliyun maven http://maven.aliyun.com/nexus/content/groups/public/ central

Configure Spark:

Tar xf spark-2.4.2.tgzcd spark-2.4.2.tgz# modifies the pom.xml file to add clouder repository cloudera https://repository.cloudera.com/artifactory/cloudera-repos/

Execute the compile command:

# execute. / dev/make-distribution.sh-- name 2.6.0-cdh6.7.0-- tgz-Pyarn-Phadoop-2.6-Phive- Phive-thriftserver-Dhadoop.version=2.6.0-cdh6.7.0 in the spark directory Note: the compilation time is about 35 minutes without any error reports. Note: the scala version used by default is the latest. If you want to specify the scala version, modify it in the following ways, such as changing the scala version to 2.10./dev/change-scala-version.sh 2.10.

Parameter description:

-- name: the suffix name of the generated package; the prefix defaults to the name of the spark version, in this example: spark-2.4.2-bin

-- tgz: the compression format is tar, and the suffix of compression is .tgz

-Pyarn: indicates that spark needs to run on yarn

-Phadoop-2.6: the id that indicates that spark uses profile of hadoop

-Dhadoop.version=2.6.0-cdh6.7.0: indicates that spark uses the version of hadoop; if not specified, the default is 2.2.0 hadoop

-Phive- Phive-thriftserver: indicates support for hive

Other parameters:

-DskipTests: skip the test

Generated files:

Under the spark directory: spark-2.4.2-bin-2.6.0-cdh6.7.0.tgz

Deploy using compiled spark:

Tar xf spark-2.4.2-bin-2.6.0-cdh6.7.0.tgzln-s spark-2.4.2-bin-2.6.0-cdh6.7.0 spark# configures the environment variable of spark [hadoop@hadoop001 ~] $vim. Bash _ profileexport SPARK_HOME=/home/hadoop/app/spark-2.4.2-bin-2.6.0-cdh6.7.0export PATH=$ {SPARK_HOME} / bin:$PATH [hadoop@hadoop001 ~] $source. Bash_profile# runs the spark test [hadoop@hadoop001 ~] $spark-shell 10:51:04 on 19-04-29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... Using builtin-java classes where applicableUsing Spark's default log4j profile: org/apache/spark/log4j-defaults.propertiesSetting default log level to "WARN" .to adjust logging level use sc.setLogLevel (newLevel). For SparkR, use setLogLevel (newLevel). Spark context Web UI available at http://hadoop001:4040Spark context available as' sc' (master = local [*] App id = local-1556506274719). Spark session available as' spark'.Welcome to _ / / _ / / _\ / _ _ / `/ _ _ / / _ /. _ _ /. _ /\ _\ version 2.4.2 / _ / Using Scala version 2.11.12 (Java HotSpot (TM) 64-Bit Server VM, Java 1.8.0 / 111) Type in expressions to have them evaluated.Type: help for more information.scala >

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.