Shulou(Shulou.com)06/03 Report--

After two days of Qingming Festival holiday, this paper summarizes two ways to use IDE to carry out spark programs. Record:

The first method is relatively simple, both of which are compiled using SBT.

Note: there is no need to install the Scala program locally, otherwise there will be version compatibility issues when compiling the program.

1. Based on Non-SBT mode

Create a Scala IDEA project

We use Non-SBT to click on "Next"

Name the project, others follow the default

Click "Finish" to complete the project creation.

Modify the properties of a project

First modify the Modules option

Create two folders under src and change their properties to source

Next, modify the Libraries

Import the jar packages needed for Spark development

After joining:

After importing the package, create a package under the scala of the project

Create an Object object

Build Spark Driver code

The program is the processing code for Sogou's log.

Next, package using Project Structure's Artifacts

Use From modules with dependencies

Select Main Class

Click "OK"

Change the name to FirstSparkAppJar

Because Scala and Spark are installed on each machine, the jar files related to Scala and Spark can be deleted

Next, Build.

Select "Build Artifacts"

Select Build for the first time, then select Rebuild for the same project later, and then wait for the compilation to be completed

Let's run the program using spark-submit

Finally, the task is completed.

Second, based on SBT mode

Development tools download

Spark development requires the following development compilation tools:

1. Scala IDE: this article takes IntelliJ IDEA as an example for development. Download it at:

Https://www.jetbrains.com/idea/download/

2. Download the SBT (Simple Build Tool) compilation tool:

Http://www.scala-sbt.org/download.html

After downloading and installing, execute the sbt command under DOS to download the required jar package:

The default jar package (.idea-build, .ivy2, .sbt) is downloaded to the user directory of disk C.

(note: to ensure network speed when downloading using the command sbt, it is best to use an agent to download)

Development tool configuration

1. IntelliJ IDEA development configuration:

(1) download the Scala plug-in: select Plugins under Configure to enter the following figure

Select Install JetBrains plugin and search Scala for download.

(2) create a Scala project based on SBT:

(3) set the project name and Scala and SBT versions:

Note:

It is best to cancel the two options of Download, otherwise the SBTjar package in the previous user directory will be overwritten, resulting in errors such as failure of compilation.

The version numbers of SBT and Scala can be seen in the C disk user directory:

C:\ Users\ user\ .sbt\ boot\ scala-2.10.4\ org.scala-sbt\ sbt\ 0.13.8

You can set it correctly at one time, or you can modify the configuration file later (build.sbt modifies scala version, build.properties modifies sbt version) to synchronize changes.

Do not use Chinese in the project path, otherwise it will not be executed even if it is compiled into a jar package.

(4) the Scala program based on SBT has strict requirements on the structure of the project, so it is necessary to establish the following directory structure:

(5) add Spark plug-in:

Note: at the same time, the plug-in needs to be added to the lib directory (just copy the operating system)

After the preparatory work is completed, let's develop the Spark program.

Write code

Here is an example of a wordcount-like program:

Write Spark programs:

Please do not use InteiiJ IDEA to compile (Chinese characters cause later compilation to fail)

Compile and execute

(1) use SBT to compile and package:

Enter the project directory under DOS, and use the sbt command to compile and package:

The default jar is under the project directory test\ target\ scala-2.10

(2) upload the jar package to the server to execute:

Use the command:

Spark-submit-class test--master yarn test_2.10-0.1-SNAPSHOT.jar 100

For more parameters, see the official documentation.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.