Shulou(Shulou.com)06/01 Report--

Editor to share with you what to pay attention to running the spark program under windows, I believe that most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to know it!

You can also run the spark program under windows. If there is no cluster, use the local mode [local], and then run the jar package.

The general steps are as follows:

(1) create a maven project that supports scala. You can write spark programs with scala. Note that spark.setMaster ("local") is like this.

(2) generate jar package. Use the mvn clean scala:compile package command.

(3) put hadoop.dll (search on the Internet) in the c:\ windows\ system32\ directory, otherwise the file created under windows will report an error.

(4) execute java-classpath yourSparkApp.jar yourMainClass

Note that if there is no spark environment, do not add words such as provided to maven, so that the jar package in the same directory will generate a lib directory, which contains all the jar packages needed by spark.

By default, when executing the jar package, you will find the jar package in the lib directory under the .jar file as a dependency.

In this way, a complete process of developing and debugging spark programs under windows is completed!

These are all the contents of this article entitled "what are the points for attention in running the spark program under windows?" Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.