Record my spark source code reading spark

Record my spark source code reading spark

Knowledge points:

1. Seq: list, suitable for scenarios such as storing ordered duplicate data and quickly inserting / deleting elements

Set is a collection, which is suitable for storing unordered and non-duplicated data, quickly finding massive elements and other scenarios.

MemoryKeys = Seq ()

2. Private declares variables to be private

Privateval hadoopConf = SparkHadoopUtil.get.newConfiguration (conf)

Private [this] val value

Private [spark], declares that variables are private and is visible in the package spark, where [spark] is a protective scope.

This is the class declaration for Master, also in the master field

Private [master] class Master (

3. Functions without parentheses

Here is a function call

Hellotheworld / / No parameters, no parentheses

4. Functions with variable parameters

(SRV *) = {s.foreach (x = > (x))}

5. the bizarre definition of function:

(XPRANY:): = XPROYY = (XRAPULY:) = > XPRONY (x:) (y:): = XPROY anonymous= (XRAPRY:) = > XPROY

6. Import: it can appear anywhere, up to the end of the block containing the statement, with the advantage of avoiding name conflicts.

SparkConf (loadDefaults: Boolean) Cloneable Logging {SparkConf._

7. This: the last sentence of the function

SetExecutorEnv (variables: Seq [(StringString)]): SparkConf = {(kv) case LOCAL_N_REGEX (threads) = > case LOCAL_N_FAILURES_REGEX (threads, maxFailures) = > case SPARK_REGEX (sparkUrl) = > case LOCAL_CLUSTER_REGEX (numSlaves, coresPerSlave, memoryPerSlave) = > case "yarn-standalone" | "yarn-cluster" = > case "yarn-client" = > case SIMR_REGEX (simrUrl) = > case mesosUrl @ MESOS_REGEX (_) = >

It can be seen that there are the above configuration types.

9. Parallelize implementation of scala

The first RDD production mode is generated from the local collection, and you can see that it is of the ParallelCollectionRDD type.

Parallelize [T: ClassTag] (seq: Seq [T] numSlices: Int = defaultParallelism): RDD [T] = withScope {assertNotStopped () ParallelCollectionRDD [T] (seqnumSlicesMap [IntSeq [string]] ())}