Shulou(Shulou.com)05/31 Report--

This article will explain in detail how to configure spark1.4.1 for you. The editor thinks it is very practical, so I share it with you as a reference. I hope you can get something after reading this article.

1. Create script

Cd / opt/spark-1.4.1-bin-hadoop2.6/confcp spark-env.sh.template spark-env.shcp slaves.template slaves

two。 Add environment variables to the program

Vi spark-env.shexport JAVA_HOME=/opt/jdk1.7.0_75export SCALA_HOME=/opt/scala-2.11.6export HADOOP_CONF_DIR=/opt/hadoop-2.6.0/etc/hadoop# spark's work directory temporary files are automatically cleaned, and every half hour export SPARK_WORKER_DIR= "/ home/hadoop/spark/worker/" export SPARK_WORKER_OPTS= "- Dspark.worker.cleanup.enabled=true-Dspark.worker.cleanup.interval=1800" vi slaves is filled in each node hostname.

3. The system adds environment variables

Vi / etc/profileexport SPARK_HOME=/opt/spark-1.4.1-bin-hadoop2.6export PATH=$SPARK_HOME/bin:$PATH

4. Start

Cd.. / sbin/./start-all.sh

5. Check to see if the process starts

Jps4211 Master4367 Worker

6. Go to the web page of spark http://spore:8080/

7. Use spark-shell

Cd.. / bin/./spark-shell

8.sparkUI http://spore:4040

Read the source code to see which sql keywords are supported by spark:

Spark\ sql\ catalyst\ src\ main\ scala\ org\ apache\ spark\ sql\ catalyst\ SQLParser.scala

Example of spark-sql custom function

Http://colobu.com/2014/12/11/spark-sql-quick-start/

If you want to use the command bin/spark-sql

Hive metastore must be started and hive.metastore.uris must be configured within the conf/hive-site.xml, for example

Hive.metastore.uris thrift://byd0087:9083

When you start bin/spark-sql, you can use hive's HQL statement, which is much faster than hive

This is the end of the article on "how to configure spark1.4.1". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.