Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to build a Spark development environment for Windows Eclipse, Xiaobian feels quite practical, so share it with you as a reference, I hope you can gain something after reading this article.

1. Spark development environment construction under Java

1.1 jdk installation

Install jdk under oracle. I installed jdk 1.7. After installing the new system environment variable JAVA_HOME, the variable value is "C:\Program Files\Java\jdk1.7.0_79", depending on the installation route.

Also add C:\Program Files\Java\jdk1.7.0_79\bin and C:\Program Files\Java\jre7\bin under the system variable Path.

1.2 Spark environment variable configuration

Go to http://spark.apache.org/downloads.html to download the corresponding version of hadoop. I downloaded spark-1.6.0-bin-hadoop2.6.tgz, spark version 1.6, corresponding to hadoop version 2.6.

Unzip the downloaded file, assuming the directory is D:\spark-1.6.0-bin-hadoop 2.6. Add D:\spark-1.6.0-bin-hadoop2.6\bin to the system Path variable, and create a new SPARK_HOME variable with the value D:\spark-1.6.0-bin-hadoop2.6

1.3 hadoop toolkit installation

Spark is based on hadoop, the relevant hadoop library will be called during the running process, if the relevant hadoop running environment is not configured, the relevant error message will be prompted, although it does not affect the operation, but here it is still hadoop related library configuration.

1.3.1 Go download hadoop 2.6, I downloaded hadoop-2.6.0.tar.gz,

1.3.2 Extract the downloaded folder, add the relevant libraries to the system Path variable: D:\hadoop-2.6.0\bin; at the same time, create a new HADOOP_HOME variable with the variable value: D:\hadoop-2.6.0, download the Windows version of winutils, add winutils.exe to your hadoop-x.x./bin, and put hadoop.dll under C:/wondwos/system32.

1.4 eclipse environment

Directly create a java project, add spark-assembly-1.6.0-hadoop2.6.0.jar under D:\spark-1.6.0-bin-hadoop2.6\lib to the project.

2. Build Spark development environment under Python

2.1 Install python and add it to the system variable path: C:\Python27 and C:\Python27\Scripts

2.2 Repeat steps 1.2 and 1.3

2.3 Copy the pyspark folder (D:\spark-1.6.0-bin-hadoop2.6\python\pyspark) from the spark directory to the python installation directory C:\Python27\Lib\site-packages

2.4 Run pip install py4j from cmd (pip must be installed first) to install the libraries.

Note: After configuring the environment variables, we program in eclipse. Take python as an example, we will report an error:

File "test.py", line 10, in conf=(SparkConf().setMaster('local').setAppName('a').setSparkHome('/home/dirk/spark-1.4.1-bin-hadoop2.6/bin')) File "/home/dirk/spark-1.4.1-bin-hadoop2.6/python/pyspark/conf.py", line 104, in __init__ SparkContext._ ensure_initialized() File "/home/dirk/spark-1.4.1-bin-hadoop2.6/python/pyspark/context.py", line 229, in _ensure_initialized SparkContext._ gateway = gateway or launch_gateway() File "/home/dirk/spark-1.4.1-bin-hadoop2.6/python/pyspark/java_gateway.py", line 48, in launch_gateway SPARK_HOME = os.environ["SPARK_HOME"] File "/usr/lib/python2.7/UserDict.py", line 23, in __getitem__ raise KeyError(key)KeyError: 'SPARK_HOME'

This is because eclipse doesn't get the configured environment variables, we can do it by

print os.getenv('SPARK_HOME')

Check, at this time we can restart the computer; if you do not want to restart the computer, we can pass

os.environ ['SPARK_HOME'] ='/home/dirk/spark-1.4.1-bin-hadoop2.6'

Set SPARK_HOME manually so that our program will work properly.

About "windows eclipse how to build spark development environment" this article is shared here, I hope the above content can be of some help to everyone, so that you can learn more knowledge, if you think the article is good, please share it to let more people see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.