Shulou(Shulou.com)06/02 Report--

This article mainly explains "how Mac deploys spark2.4.4". The explanation in this article is simple and clear, easy to learn and understand. Please follow the ideas of Xiaobian and go deep into it slowly to study and learn "how Mac deploys spark2.4.4" together!

environmental information

Operating System: macOS Mojave 10.14.6

JDK: 1.8.0_211 (installed in/Library/Java/JavaVirtualMachines/jdk1.8.0_211.jdk/Contents/Home)

prerequisite

Please refer to the article "Mac deployment hadoop3(pseudo-distributed)" to deploy hadoop3 on Mac in advance

deployment steps

Install Scala:

brew install scala

Configure scala environment variables, open the file ~/.bash_profile, and add the following configuration:

export SCALA_HOME=/usr/local/Cellar/scala/2.13.0export PATH=$PATH:$SCALA_HOME/bin

Execute the command source ~/.bash_profile and verify scala:

base) zhaoqindeMBP:~ zhaoqin$ scala -versionScala code runner version 2.13.0 -- Copyright 2002-2019, LAMP/EPFL and Lightbend, Inc.

Download spark, address: spark.apache.org/downloads.html, as shown in the red box below:

Unzip the downloaded files in/usr/local/and change the folder name from spark-2.4.4-bin-hadoop2.7 to spark

Configure the environment variables of spark, open the file ~/.bash_profile, and add the following configuration contents:

export SPARK_HOME=/usr/local/sparkexport PATH=$PATH:$SPARK_HOME/bin

Execute the command source ~/.bash_profile to make the configuration effective;

Open the file spark/conf/spark-env.sh and add the following three lines to the end:

export SCALA_HOME=/usr/local/Cellar/scala/2.13.0export SPARK_MASTER_IP=localhostexport SPARK_WORKER_MEMORY=2G

Make sure hdfs and yarn are started, and then execute the command spark-shell to start the spark service:

To update your account to use zsh, please run `chsh -s /bin/zsh`.For more details, please visit https://support.apple.com/kb/HT208050. (base) zhaoqindeMBP:~ zhaoqin$ spark-shell19/10/27 13:33:51 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableUsing Spark's default log4j profile: org/apache/spark/log4j-defaults.propertiesSetting default log level to "WARN".To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).Spark context Web UI available at http://zhaoqindembp:4040Spark context available as 'sc' (master = local[*], app id = local-1572154437623).Spark session available as 'spark'.Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.4.4 /_/Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_211)Type in expressions to have them evaluated.Type :help for more information.scala> Thank you for reading, the above is the content of "How to deploy spark2.4.4 on Mac". After studying this article, I believe everyone has a deeper understanding of how to deploy spark2.4.4 on Mac. The specific usage needs to be verified by practice. Here is, Xiaobian will push more articles related to knowledge points for everyone, welcome to pay attention!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.