Shulou(Shulou.com)06/03 Report--

In fact, the most important thing in the last article is the setting of JAVA environment variable and hadoop environment variable. If these two settings are set, there will be no problem with running hadoop.

It's easy to install spark on top of hadoop.

I need to install the Hadoop cluster before installing Spark, because I already installed hadoop, so I installed spark directly on the previous hadoop cluster.

Hardware environment:

Hddcluster1 10.0.0.197 redhat7

Hddcluster2 10.0.0.228 centos7 this one as master

Hddcluster3 10.0.0.202 redhat7

Hddcluster4 10.0.0.181 centos7

Software environment:

Scala-2.11.7

Spark-2.0.2-bin-hadoop2.7.tgz

# all operating functions hadoop

Basic process:

1. Master decompresses scala-2.11.7 and spark-2.0.2-bin-hadoop2.7.tgz to the appropriate directory

2. Configure scala and spark environment variables

3. Modify the configuration file

4. Copy scala and spark to each node and authorize

5. Start the spark cluster

# hadoop user actions Download scala Install wget http://downloads.lightbend.com/scala/2.11.7/scala-2.11.7.tgztar-zxvf scala-2.11.7.tgz mv scala-2.11.7 / usr/local/scalasudo mv scala-2.11.7 / usr/local/scalavim .bashrc # add export SCALA_HOME=/usr/local/scalaexport PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$SCALA_HOME/binsource .bashrc [hadoop@hddcluster2 ~] $scala- versionScala code runner version 2.11.7-- Copyright 2002-2013 LAMP/EPFL# downloads spark-2.0.2-bin-hadoop2.7.tgztar-zxvf spark-2.0.2-bin-hadoop2.7.tgzmv spark-2.0.2-bin-hadoop2.7 sparksudo mv spark/ usr/local/vim .bashrc # add export SPARK_HOME=/usr/local/sparkexport PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$SCALA_HOME/bin:$SPARK_HOME# and modify spark configuration file cd / usr/local/spark/confcp spark on the official website -env.sh.template spark-env.shvi spark-env.sh # add the following # jdk direxport JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-2.b15.el7_3.x86_64###scala direxport SCALA_HOME=/usr/local/scala###the ip of master node of sparkexport SPARK_MASTER_IP=10.0.0.228###the max memory size of workerexport SPARK_WORKER_MEMORY=8G### Hadoop configuration file direxport HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop/# modifies slavecp slaves.template slavesvi slaves # change localhost to the following hddcluster1hddcluster2hddcluster3hddcluster4# package / usr/local/spark and / usr/local/scala Then copy to the slave node, Cd / usr/localtar-zcf ~ / master.spark.tar.gz. / sparktar-zcf ~ / master.scala.tar.gz. / scalascp master.spark.tar.gz hddcluster1:~scp master.scala.tar.gz hddcluster1:~# log in to each node and decompress to / usr/localtar-zxf master.spark.tar.gz-C / usr/local/tar-zxf master.scala.tar.gz-C / usr/local/chown-R hadoop:hadoop / usr/local/sparkchown-R hadoop: Hadoop / usr/local/scala reconfigures the .bashrc environment variable as master. Plus the .bashrc content of the previous hadoop article goes like this: # scalaexport SCALA_HOME=/usr/local/scala#sparkexport SPARK_HOME=/usr/local/spark#java and hadoopexport JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-2.b15.el7_3.x86_64export HADOOP_HOME=/usr/local/hadoopexport HADOOP_INSTALL=$HADOOP_HOMEexport HADOOP_MAPRED_HOME=$HADOOP_HOMEexport HADOOP_COMMON_HOME=$HADOOP_HOMEexport HADOOP_HDFS_HOME=$HADOOP_HOMEexport YARN_HOME=$HADOOP_ HOMEexport HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/nativeexport PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$SCALA_HOME/bin:$SPARK_HOMEexport HADOOP_PREFIX=$HADOOP_HOMEexport HADOOP_OPTS= "- Djava.library.path=$HADOOP_PREFIX/lib:$HADOOP_PREFIX/lib/native" to this Spark cluster has been built. Start Spark cluster: you need to start dfs and yarn of hadoop before starting Spark. After / usr/local/spark/sbin/start-all.sh starts all services, enter jps: [hadoop@hddcluster2 ~] $jps29601 ResourceManager32098 SparkSubmit29188 DataNode29364 SecondaryNameNode29062 NameNode29915 NodeManager30251 Master30380 Worker30062 JobHistoryServer18767 Jps on the command line with more Master and worker/usr/local/spark/bin/spark-shell.sh than when the hadoop cluster starts. When scala > appears, it is successful. When you type 10.0.0.228 Worker 8080 in the browser, you will see that there are four RMOs as shown in the figure below.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.