How to use java to submit a Spark application
This article introduces the knowledge of "how to submit a Spark application with java". Many people will encounter this dilemma in the operation of actual cases, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!
The first way
First open a file with vim, MyLauncher.java
The code is as follows:
Import org.apache.spark.launcher.SparkAppHandle
Import org.apache.spark.launcher.SparkLauncher
Import java.util.HashMap
Public class MyLauncher {
Public static void main (String [] args) throws Exception {
HashMap map = newHashMap ()
Map.put ("HADOOP_CONF_DIR", "/ home/hadoop/conf")
Map.put ("YARN_CONF_DIR", "/ home/hadoop/conf")
Map.put ("SPARK_CONF_DIR", "/ home/hadoop/spark/conf")
New SparkLauncher (map)
.setAppResource ("/ data/newStreaming/uesc-analyzer.jar")
.setMainClass ("ucloud.UESBash.testSchema")
.setMaster ("yarn-cluster")
.setConf (SparkLauncher.DRIVER_MEMORY, "2g")
.setVerbose (true) .startApplication ()
Thread.sleep (100000)
/ / Use handle API to monitor / control application.
}
}
Next, compile
Javac-cp / home/hadoop/spark/lib/spark-assembly-1.6.0-hadoop2.6.0-cdh6.4.9.jarMyLauncher.java
Then submit it for execution.
Java-cp/home/hadoop/spark/lib/spark-assembly-1.6.0-hadoop2.6.0-cdh6.4.9.jar:.MyLauncher
So you can see the operation on the interface of yarn.
Note: you may be wondering why I added a pause there, because normally we have to do some other actions to monitor the Spark application instead of quitting directly, which may cause the submission to be completed and exit, so that the application cannot be submitted.
Of course, there is another way, but it is not recommended. The method used above can better monitor the status of our Spark programs.
The second way
Import org.apache.spark.launcher.SparkAppHandle
Import org.apache.spark.launcher.SparkLauncher
Import java.util.HashMap
Public class MyLauncher {
Public static void main (String [] args) throws Exception {
HashMap map = newHashMap ()
Map.put ("HADOOP_CONF_DIR", "/ home/hadoop/conf")
Map.put ("YARN_CONF_DIR", "/ home/hadoop/conf")
Map.put ("SPARK_CONF_DIR", "/ home/hadoop/spark/conf")
Process spark = new SparkLauncher (map)
.setAppResource ("/ data/newStreaming/uesc-analyzer.jar")
.setMainClass ("ucloud.UESBash.testSchema")
.setMaster ("yarn-cluster")
.setConf (SparkLauncher.DRIVER_MEMORY, "2g")
.setVerbose (true) .launch ()
/ / Use handle API to monitor / control application.
Spark.waitFor ()
}
}
The second way is to start a child process to load the submission application.
That's all for "how to submit a Spark application with java". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!
