Shulou(Shulou.com)06/01 Report--

Editor to share with you how to build the spark1.4 Windows local debugging environment, I believe most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to know it!

Summary on the Construction of spark1.4 Windows local debugging Environment

1.scala version of scala-2.10.4 officially recommended scala-2.11.7 [not recommended, non-sbt projects, load after need]

2.spark version of spark-1.4.0-bin-hadoop2.6.tgz

3.hadoop version 3.1 hadoop-2.6.0.tar.gz 3.2 environment variable HADOOP_HOME=E:/ysg.tools/spark/hadoop-2.6.0 or System.setProperty ("hadoop.home.dir", "E:\ ysg.tools\ spark\ hadoop-2.6.0"); 3.3winutils.exe

Copy winutils.exe to spark/hadoop-2.6.0/bin

File download address http://files.cnblogs.com/files/yjmyzz/hadoop2.6%28x64%29V0.2.zip

4.idea New NO-SBT Project

Libraties adds scala sdk spark-1.4.0-bin-hadoop2.6\ lib\ spark-assembly-1.4.0-hadoop2.6.0.jar

Spark.test.iml loads spark-assembly-1.4.0-hadoop2.6.0 first and then scala-sdk-2.11.7

These are all the contents of this article entitled "how to build a spark1.4 Windows local debugging Environment". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.