Shulou(Shulou.com)06/01 Report--

This article is to share with you the example analysis of macOS Spark 2.4.3 standalone building, the editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

Based on

Jdk 1.8

Mac os

1. Decompression

Nancylulululu:local nancy$ tar-zxvf / Users/nancy/Downloads/spark-2.4.3-bin-hadoop2.7.tar-C / usr/local/opt/

2. Rename and modify the configuration file

Nancylulululu:conf nancy$ cp slaves.template slaves

Nancylulululu:conf nancy$ cp spark-defaults.conf.template spark-defaults.conf

Nancylulululu:conf nancy$ cp spark-env.sh.template spark-env.sh

Nancylulululu:conf nancy$ vi slaves

Localhost

Nancylulululu:conf nancy$ vi spark-env.sh

SPARK_LOCAL_IP=127.0.0.1

SPARK_MASTER_HOST=127.0.0.1

SPARK_MASTER_PORT=7077

SPARK_WORKER_CORES=2

SPARK_WORKER_MEMORY=1G

3. Start spark

Nancylulululu:spark-2.4.3 nancy$ sbin/start-all.sh

Starting org.apache.spark.deploy.master.Master, logging to / usr/local/opt/spark-2.4.3/logs/spark-nancy-org.apache.spark.deploy.master.Master-1-nancylulululu.out

Localhost: starting org.apache.spark.deploy.worker.Worker, logging to / usr/local/opt/spark-2.4.3/logs/spark-nancy-org.apache.spark.deploy.worker.Worker-1-nancylulululu.out

Jps views a master and a worker service

Nancylulululu:conf nancy$ jps

3908 CoarseGrainedExecutorBackend

3478 Master

4166 Jps

3896 SparkSubmit

3514 Worker

4 、 Launching Applications with spark-submit

Once a user application is bundled, it can be launched using the bin/spark-submit script. This script takes care of setting up the classpath with Spark and its dependencies, and can support different cluster managers and deploy modes that Spark supports:

. / bin/spark-submit\-- class\-- master\-- deploy-mode\-- conf =. # other options\ [application-arguments]

Some of the commonly used options are:

-- class: The entry point for your application (e.g. org.apache.spark.examples.SparkPi)

-- master: The master URL for the cluster (e.g. spark://23.195.26.187:7077)

-- deploy-mode: Whether to deploy your driver on the worker nodes (cluster) or locally as an external client (client) (default: client)

Conf: Arbitrary Spark configuration property in key=value format. For values that contain spaces wrap "key=value" in quotes (as shown).

Application-jar: Path to a bundled jar including your application and all dependencies. The URL must be globally visible inside of your cluster, for instance, an hdfs:// path or a file:// path that is present on all nodes.

Application-arguments: Arguments passed to the main method of your main class, if any

Script:

. / bin/spark-submit\-class org.apache.spark.examples.SparkPi\-master spark://localhost:7077\ / usr/local/opt/spark-2.4.3/examples/jars/spark-examples_2.11-2.4.3.jar\ 100

Result

Pi is roughly 3.1413047141304715

14:23:06 on 19-05-14 INFO SparkUI: Stopped Spark web UI at http://localhost:4040

5. Spark-shell mode

Nancylulululu:spark-2.4.3 nancy$ bin/spark-shell-master spark://localhost:7077

6. Test wordcount

Test file

Nancylulululu:opt nancy$ vi 3.txt

Hello scala hello spark hello mysql hello java hello java hello scala

Test result

Scala > sc.textFile ("/ usr/local/opt/3.txt"). FlatMap (_ .split (")). Map ((_, 1)). ReduceByKey (_ + _). Collect

Res1: Array [(String, Int)] = Array ((scala,2), (mysql,1), (hello,6), (java,2), (spark,1))

Scala >

The above is the example analysis built by macOS Spark 2.4.3 standalone. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.