Shulou(Shulou.com)06/01 Report--

This article introduces you how to install Spark in the centOS7 system, the content is very detailed, interested friends can refer to, hope to be helpful to you.

Environment description:

Operating system: centos7 64-bit 3 sets

Centos7-1 192.168.190.130 master

Centos7-2 192.168.190.129 slave1

Centos7-3 192.168.190.131 slave2

Installing spark requires the following to be installed at the same time:

Jdk scale

1. Install jdk and configure jdk environment variables

Here is not about how to install and configure jdk, own Baidu.

two。 Install scala

# tar-zxvf scala-2.13.0-M4.tgz modify the / etc/profile file again to add the following: export SCALA_HOME=$WORK_SPACE/scala-2.13.0-M4 export PATH=$PATH:$SCALA_HOME/bin # source / etc/profile / / make it effective immediately # scala- version / / check whether the installation of scala is complete

3. Install spark

Source code: Spark source code, which needs to be compiled to use. In addition, Scala 2.11 needs to be compiled with source code before it can be used.

Pre-build with user-provided Hadoop: "Hadoop free" version, which can be applied to any Hadoop version

Pre-build for Hadoop 2.7and later: based on the pre-compiled version of Hadoop 2.7, it needs to correspond to the version of Hadoop installed locally. Hadoop 2.6 is also optional. Because the hadoop installed here is 3.1.0, I directly install the version of for hadoop 2.7 and later.

Note: for the installation of hadoop, please check my previous blog. I will not repeat the description.

Install and configure Spark under centOS7 # mkdir spark# cd / usr/spark#tar-zxvf spark-2.3.1-bin-hadoop2.7.tgz#vim / etc/profile# add spark environment variables, such as under PATH, export out # source / etc/profile# and enter the conf directory Make a copy of spark-env.sh.template and rename it spark-env.sh#cd / usr/spark/spark-2.3.1-bin-hadoop2.7/conf#cp spark-env.sh.template spark-env.sh#vim spark-env.shexport SCALA_HOME=/usr/scala/scala-2.13.0-M4export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.171-8.b10.el7_5.x86_64export HADOOP_ HOME=/usr/hadoop/hadoop-3.1.0export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoopexport SPARK_HOME=/usr/spark/spark-2.3.1-bin-hadoop2.7export SPARK_MASTER_IP=masterexport SPARK_EXECUTOR_MEMORY=1G# enters the conf directory Copy a copy of slaves.template to be renamed slaves#cd / usr/spark/spark-2.3.1-bin-hadoop2.7/conf#cp slaves.template slaves#vim slaves# add node domain name to slaves file # master / / the domain name is centos7-1 # slave1 / / the domain name is centos7-2 # slave2 / / the domain name is centos7-3

Start spark

# start the hadoop node before starting spark # cd / usr/hadoop/hadoop-3.1.0/#sbin/start-all.sh#jps / / check whether the started thread has started the hadoop # cd / usr/spark/spark-2.3.1-bin-hadoop2.7#sbin/start-all.sh Note: spark must also be installed on the slave1\ slave2 node as above Or copy a copy directly to the slave1,slave2 node # scp-r / usr/spark root@slave1ip:/usr/spark

The startup information is as follows:

Starting org.apache.spark.deploy.master.Master, logging to / usr/spark/logs/spark-root-org.apache.spark.deploy.master.Master-1-master.out

Slave2: starting org.apache.spark.deploy.worker.Worker, logging to / usr/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave2.com.cn.out

Slave1: starting org.apache.spark.deploy.worker.Worker, logging to / usr/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave1.com.cn.out

Master: starting org.apache.spark.deploy.worker.Worker, logging to / usr/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-master.out

Test the Spark cluster:

Open the spark cluster url: http://192.168.190.130:8080/ on the master node with a browser

On how to install Spark in the centOS7 system to share here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.