Shulou(Shulou.com)06/01 Report--

Editor to share with you the example analysis of spark jobserver source code, I believe that most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to know it!

Characteristics of spark jobserver:

1. "Spark as Service": REST-style api interface is provided for all aspects of job and contexts for management 2. Support for SparkSQL, Hive, StreamingContext/jobs and custom job contexts! 3. By integrating Apache Shiro to support LDAP rights verification 4. In order to isolate each sparkcontext, a separate jvm5. Support for subsecond-level low-latency tasks 6. 5 through long-running job contexts. You can stop a running job (job) 7. 0 by ending context. Split the jar upload steps to improve the startup speed of job by 8. 0. Asynchronous and synchronous job API, in which synchronous API is very effective for low latency jobs. Support Standalone Spark and Mesos, yarn10. Job and jar messages are persisted through a pluggable DAO interface. Name and cache the RDD or DataFrame object, and get the RDD or DataFrame by that name. This improves the sharing and reuse of objects among jobs.

It is recommended that you start from the bin directory of Spark jobserver, such as the server_start.sh script, which itself is a good example of a spark task submission script.

#! / usr/bin/env bash# Script to start the job server# Extra arguments will be spark-submit options, for example#. / server_start.sh-- jars cassandra-spark-connector.jar## Environment vars (note settings.sh overrides): # JOBSERVER_MEMORY-defaults to 1G, the amount of memory (eg 512m, 2G) to give to job server# JOBSERVER_CONFIG-alternate configuration file to use# JOBSERVER_FG-launches job server in foreground; defaults to forking in backgroundset-e

Get_abs_script_path () {pushd. > / dev/null cd "$(dirname" $0 ")" appdir=$ (pwd) popd > / dev/null}

Get_abs_script_path

Set-a. $appdir/setenv.shset + a

GC_OPTS_SERVER= "$GC_OPTS_BASE-Xloggc:$appdir/$GC_OUT_FILE_NAME"

MAIN= "spark.jobserver.JobServer"

PIDFILE=$appdir/spark-jobserver.pidif [- f "$PIDFILE"] & & kill-0 $(cat "$PIDFILE"); then echo 'Job server is already running' exit 1fi

Cmd='$SPARK_HOME/bin/spark-submit-- class $MAIN-- driver-memory $JOBSERVER_MEMORY-- conf "spark.executor.extraJavaOptions=$LOGGING_OPTS"-- driver-java-options "$GC_OPTS_SERVER $JAVA_OPTS_SERVER $LOGGING_OPTS $CONFIG_OVERRIDES" $@ $appdir/spark-job-server.jar $conffile'if [- z "$JOBSERVER_FG"]; then eval $cmd > $LOG_DIR/server_start.log 2 > & 1

$PIDFILEelse eval $cmdfi

You can see the familiar spark-submit, the entry function is the main function of spark.jobserver.JobServer, and the tip of the wave is not expanded here.

The general structure of spark jobserver is shown in the following picture, you can also refer to the ideas shown in the following figure to read, ball players can also go to the planet to find my weekly tips.

The above is all the contents of this article "sample Analysis of spark jobserver Source Code". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.