Shulou(Shulou.com)05/31 Report--

What this article shares with you is a discussion about how to deploy a Spark cluster. The editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

Spark cluster deployment

Step 1: install the software

Spark 1.5.4:wget http://www.apache.org/dyn/closer.lua/spark/spark-1.5.2/spark-1.5.2-bin-hadoop2.6.tgz

Hadoop 2.6.3:wget http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.6.3/hadoop-2.6.3.tar.gz

Scala: apt-get install scala

Step 2: configure environment variables

Execute / etc/profile

Export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_80

Export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/BIN:$ {SCALA_HOME} / bin:$ {SPARK_HOME} / bin:/home/supermap/program/hadoop-2.6.3/bin:$PATH

Export CLASSPATH=$CLASSPATH:.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib

Export SCALA_HOME=/usr/lib/scala/scala-2.11.7

Export PATH=$ {SCALA_HOME} / bin:$PATH

Export SPARK_HOME=/program/spark-1.5.2-bin-hadoop2.6

Source / etc/profile

Step 3: create a mirror

Create an image based on the configured virtual machine through OpenStack Manager, and the resulting image configuration is as follows:

Mirror profile information

_ _

Name spark-hadoop

ID 61055db5-598b-4f1a-98fa-d2cbbf305d0c

Status Active

Public False

Protected False

Checksum 47acf7993101713aee17764802602941

_ _

Configure 4.4 GB

Container format BARE

Disk format QCOW2

Minimum disk 60.0GB

Step 4: create a virtual machine

Based on the created image, two virtual machines are generated, plus one that was originally used to create the image, a total of three virtual hosts are named:

Spark_hadoop_master

Spark_hadoop_slave1

Spark_hadoop_slave2

Now you are ready to do something related to creating a Spark cluster.

Step 5: set the hostname

1. Change the hostname in / etc/hostname, set the three hosts to master, slave1, slave2, and restart

By testing root@master: ~ # ssh slave1, you can log in after entering the root user password.

Step 6: modify ssh password-less access

First, enable root user access rights.

1. Modify the / etc/ssh/sshd-config file with the following settings:

2. PermitRootLogin yes

3. PubkeyAuthentication yes

4. PasswordAuthentication yes

Restart the ssh service: service ssh restart

Then do the following test, you can log in without entering a password.

Copy spark-env.sh.template to spark-env.sh

Copy slaves.template to slaves

Vim opens spark-env.sh to modify its contents and adds the following settings

Export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_80

Export SPARK_MASTER_IP=192.168.12.154

Export SPARK_WORKER_MEMORY=4g

Export SCALA_HOME=/usr/lib/scala/scala-2.11.7

Export HADOOP_CONF_DIR=/home/supermap/program/hadoop-2.6.3/conf

SPARK_WORKER_MEMORY: the defined Worker node can maximize the amount of memory allocated to the Excutors. Since the configured virtual machine is 4G memory, in order to maximize memory use, set 4G here.

Next, configure the slaves file to add all the Workers nodes:

Finally, check the service status by visiting the spark management page

The above is how to carry out the discussion of Spark cluster deployment, the editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.