Shulou(Shulou.com)06/01 Report--

This article mainly introduces how to install shark, has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let the editor take you to understand it.

Spark0.9.1 installation

Decompress

Modify directory user rights, user hadoop login

Modify the configuration file

/ / slaves

Hadoop1

Hadoop2

Hadoop3

/ / spark-env.sh

Export SPARK_MASTER_IP=hadoop1

Export SPARK_MASTER_PORT=7077

Export SPARK_WORKER_CORES=1

Export SPARK_WORKER_INSTANCES=1

Export SPARK_WORKER_MEMORY=3g

Scp to a node

* * *

* * *

Shark installation

Download source code and decompress

Source code compilation

SHARK_HADOOP_VERSION=2.2.0 SHARK_YARN=true sbt/sbt assembly

Modify the configuration file

/ / shark-env.sh

Export SPARK_MEM=2g

# (Required) Set the master program's memory

Export SHARK_MASTER_MEM=1g

# (Optional) Specify the location of Hive's configuration directory. By default

# Shark run scripts will point it to $SHARK_HOME/conf

Export HIVE_CONF_DIR= "/ app/hadoop/hive013/conf"

# For running Shark in distributed mode, set the following:

Export HADOOP_HOME= "/ app/hadoop/hadoop220"

Export SPARK_HOME= "/ app/hadoop/spark091"

Export MASTER= "spark://hadoop1:7077"

Scp to a node

* * *

* * *

Start hive metasotre service

Nohup bin/hive-- service metastore > metastore.log 2 > & 1 &

Start shark

Bin/shark

. / bin/shark-service sharkserver

. / bin/shark-h-p

Thank you for reading this article carefully. I hope the article "how to install shark" shared by the editor will be helpful to everyone. At the same time, I also hope that you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.