Shulou(Shulou.com)06/03 Report--

There are a total of three machines in the cluster, which are run by hadoop users. The three machines are as follows:

1DEV-HADOOP-01192.168.9.1102DEV-HADOOP-02192.168.9.1113DEV-HADOOP-03192.168.9.112

Configure password-less login for hadoop users between clusters

Configure / etc/hosts for each machine

Install JDK1.8.0_60

Install scala

Download scala-2.11.7.tar

Extract the scala-2.11.7.tar to the / data/server directory

Set the environment variable / etc/profile

Export SCALA_HOME=/usr/scala/scala-2.11.7

Make the configuration effective source / etc/profile

Install spark

Download spark-1.5.0.tar

Extract the spark-1.5.0.tar to the / data/server directory

Configure the environment variable / etc/profile

Export SPARK_HOME=/data/server/spark-1.5.0-bin-hadoop2.6

Export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin

Make the configuration effective source / etc/profile

Modify spark configuration file

Spark-env.sh profile

Export SCALA_HOME=/data/server/scala-2.11.7

Export JAVA_HOME=/usr/java/jdk1.8.0_60/

Export SPARK_MASTER_IP=DEV-HADOOP-01

Export SPARK_WORKER_MEMORY=2048m

Slaves profile

DEV-HADOOP-02

DEV-HADOOP-03

Start the spark cluster

Sbin/start-all.sh

Cluster startup verification

By executing the jps command, the master node can see the Master process, and the slave can see the Worker process

View Spark UI

Http://192.168.9.110:8080/

Shut down the cluster

Sbin/stop-all.sh

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.