Shulou(Shulou.com)06/01 Report--

This article mainly introduces "how to build a Spark source reading environment". In daily operation, I believe many people have doubts about how to build a Spark source reading environment. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful to answer the questions of "how to build a Spark source reading environment". Next, please follow the editor to study!

1. Install JDK1.7, Maven3.2.5, Scala2.10.4

two。 Download and extract Spark1.4.0

Download address: http://mirrors.cnnic.cn/apache/spark/spark-1.4.0/spark-1.4.0.tgz

3. Download and install IntelliJ IDEA 14.1.4, install the Scala plug-in for IDEA, and then import the Spark project

4. Compile Spark in IDEA

If you encounter the error shown in the following figure when compiling:

Click "Generate Sources and Update Folders For All Projects" in the Maven window to solve this problem

You may encounter a "Unable to import maven project: See logs for details" error during "Generate Sources and Update Folders For All Projects". Click Help-- > Show Log in Exploere to view the error log information as follows:

2015-08-29 22 WARN WARN-ution.rmi.RemoteProcessSupport-Exception in thread "RMI TCP Connection (idle)" java.lang.OutOfMemoryError: PermGen space 2015-08-29 22 22 embedder class org.jetbrains.idea.maven.project.MavenEmbeddersManager.FOR_FOLDERS_RESOLVE is already used 08909 [105808] WARN-ution.rmi.RemoteProcessSupport-Exception in thread "RMI TCP Connection" (idle) "java.lang.OutOfMemoryError: PermGen space 2015-08-29 22 ERROR-# org.jetbrains.idea.maven-Cannot reconnect. Java.lang.RuntimeException: Cannot reconnect.at org.jetbrains.idea.maven.server.RemoteObjectWrapper.perform (RemoteObjectWrapper.java:82) at org.jetbrains.idea.maven.server.MavenEmbedderWrapper.customizeForResolve (MavenEmbedderWrapper.java:64) at org.jetbrains.idea.maven.project.MavenProjectsTree.executeWithEmbedder (MavenProjectsTree.java:1366) at org.jetbrains.idea.maven.project.MavenProjectsTree.resolveFolders (MavenProjectsTree.java:1312) at org.jetbrains.idea.maven.project.MavenProjectsProcessorFoldersResolvingTask.perform (MavenProjectsProcessorFoldersResolvingTask.java:39) at org.jetbrains.idea .maven.project.MavenProjectsProcessor.doProcessPendingTasks (MavenProjectsProcessor.java:134) at org.jetbrains.idea.maven.project.MavenProjectsProcessor.access$100 (MavenProjectsProcessor.java:30) at org.jetbrains.idea.maven.project.MavenProjectsProcessor$2.run (MavenProjectsProcessor.java:109) at org.jetbrains.idea.maven.utils.MavenUtil$6.run (MavenUtil.java:441) at com.intellij.openapi.application.impl.ApplicationImpl$8.run (ApplicationImpl.java:400) at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:471) at java.util .concurrent.FutureTask.run (FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:615) at java.lang.Thread.run (Thread.java:745) at org.jetbrains.ide.PooledThreadExecutor$1 $1.run (PooledThreadExecutor.java:56) Caused by: java.rmi.UnmarshalException: Error unmarshaling return Nested exception is: java.io.EOFExceptionat sun.rmi.transport.StreamRemoteCall.executeCall (StreamRemoteCall.java:246) at sun.rmi.server.UnicastRef.invoke (UnicastRef.java:161) at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod (RemoteObjectInvocationHandler.java:194) at java.rmi.server.RemoteObjectInvocationHandler.invoke (RemoteObjectInvocationHandler.java:148) at com.sun.proxy.$Proxy91.set (Unknown Source) at sun.reflect.GeneratedMethodAccessor124.invoke (Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java. Lang.reflect.Method.invoke (Method.java:606) at com.intellij.execution.rmi.RemoteUtil.invokeRemote (RemoteUtil.java:124) at com.intellij.execution.rmi.RemoteUtil.access$100 (RemoteUtil.java:36) at com.intellij.execution.rmi.RemoteUtil$2 $1 $1.compute (RemoteUtil.java:105) at com.intellij.execution.rmi.RemoteUtil.executeWithClassLoader (RemoteUtil.java:181) at com.intellij.execution.rmi.RemoteUtil$2 $1.invoke (RemoteUtil.java:102) at com.sun.proxy.$Proxy91.set ( Unknown Source) at org.jetbrains.idea.maven.server.MavenServerManager.create (MavenServerManager.java:175) at org.jetbrains.idea.maven.server.MavenServerManager.create (MavenServerManager.java:71) at org.jetbrains.idea.maven.server.RemoteObjectWrapper.getOrCreateWrappee (RemoteObjectWrapper.java:41) at org.jetbrains.idea.maven.server.MavenServerManager$5.create (MavenServerManager.java:481) at org.jetbrains.idea.maven.server.MavenServerManager$5.create (MavenServerManager.java:469) at org.jetbrains.idea.maven.server.RemoteObjectWrapper GetOrCreateWrappee (RemoteObjectWrapper.java:41) at org.jetbrains.idea.maven.server.MavenEmbedderWrapper.doCustomize (MavenEmbedderWrapper.java:87) at org.jetbrains.idea.maven.server.MavenEmbedderWrapper.access$000 (MavenEmbedderWrapper.java:36) at org.jetbrains.idea.maven.server.MavenEmbedderWrapper$2.execute (MavenEmbedderWrapper.java:67) at org.jetbrains.idea.maven.server.RemoteObjectWrapper.perform (RemoteObjectWrapper.java:76). 15 moreCaused by: java.io.EOFExceptionat java.io.ObjectInputStream$BlockDataInputStream.peekByte (ObjectInputStream.java:2598) at java.io.ObjectInputStream.readObject0 (ObjectInputStream.java:1318) at java.io.ObjectInputStream.defaultReadFields (ObjectInputStream.java:1990) at java.io.ObjectInputStream.readSerialData (ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject (ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0 (ObjectInputStream.java:1350) at java.io.ObjectInputStream.readObject (ObjectInputStream.java:370) at sun.rmi.transport. StreamRemoteCall.executeCall (StreamRemoteCall.java:244)... 38 more2015-08-29 22 Build 59 ERROR-# org.jetbrains.idea.maven-IntelliJ IDEA 14.1.4 Build # IU-141.1532.4 2015-08-29 22 22 Build # IU-141.1532.4 2015-08-29 22 25 ERROR-# org.jetbrains.idea.maven-JDK: 1.7.058 80 2015-08-29 22 22 Build 08924 [105823] ERROR-# org.jetbrains.idea.maven-VM: Java HotSpot (TM) 64-Bit Server VM 2015-08-29 22 Vendor ERROR-# org.jetbrains.idea.maven-Vendor: Oracle Corporation 2015-08-29 22 22 Vendor 08924 [105823] ERROR-# org.jetbrains.idea.maven-OS: Windows 82 015-08-29 22 22 Vendor 59 Last Action: ERROR-# org.jetbrains.idea.maven-Last Action: Maven.UpdateFolders

This should be the reason for GFW. Just change the time period or use an agent.

5. Run master in IDE: run the class org.apache.spark.deploy.master.Master directly, if an error is reported as shown in the following figure

Modify the pom.xml under the root path of the project, and change the version of guava from provided to compile.

Com.google.guava

Guava

14.0.1

Compile

When the operation is successful, the console output is as follows:

6. Run Worker in IDE: run the class org.apache.spark.deploy.worker.Worker and specify the parameter-- webui-port 8081 spark://192.168.63.1:7077

At this point, the study on "how to build a Spark source reading environment" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.