Shulou(Shulou.com)06/03 Report--

1. Spark runtime architecture:

Spark distributed architecture adopts master / slave structure mode. The master is the driver (Driver) node, which is responsible for central coordination and scheduling of each work (actuator executor) node.

The slave is the executor node.

Spark driver nodes and actuator nodes are collectively referred to as Spark applications. The Spark application is launched on the machine in the cluster through the cluster manager.

2. Tasks of drivers and actuators:

Drive tasks: responsible for running the tasks that make up the Spark job

Executor task: provides memory storage for RDD that requires caching.

III. Cluster Manager

Cluster Manager can be used to start drive nodes and actuator nodes. Generally speaking, the following cluster managers are included: the cluster manager that comes with Hadoop Yarn,Apache Mesos,Spark.

4. The format of spark-submit:

/ bin/spark-submit [options] [app options]

Fifth, Spark performance tuning:

How to tune and debug Spark workloads in a production environment.

1. Adjust the runtime configuration options for the Spark application. Spark is configured through the SparkConf class.

Val conf = new SparkConf ()

Conf.set ("spark.app.name", "My Spark App")

Conf.set ("spark.master", "local [4]")

Conf.set ("spark.ui.port", "36000")

Val sc = new SparkContext (conf)

two。 When submitting a task to spark-submit, you can use the-- conf option.

For example, bin/spark-submit-class com.vip.SimpleClass-master local [4]-name "My Spark App"-conf spark.ui.port=36000 myapp.jar

3. Specify the path to the configuration file through the-- properties-file flag of spark-submit:

Spark-submit-class com.vip.SimpleClass-properties-file my-config.conf

Since there are three different ways to set parameters, if a conflict occurs after all three cases are configured, there is a priority difference between the three ways. The priority from high to low is 1, 1, 2 and 3, respectively. In the event of conflict, the first type shall prevail.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.