Shulou(Shulou.com)06/03 Report--

The operation modes supported by spark: local mode, local cluster mode, standalone mode, yarn mode and mesos mode.

Local mode

Local, local [N] or local [NMagneMaxRetries]. Mainly used for code debugging and tracking. Do not have fault tolerance, not suitable for production environment.

The local mode is only Driver, not Master and Worker. The Executor that executes the task is in the same JVM process as Driver.

Local cluster mode

Local-cluster [N,cores,memory] . It is also mainly used for code debugging and testing, and it is a common mode for source code learning. It does not have fault tolerance and cannot be used in production environment.

Driver, Master, and Worker run in the same JVM process. Each Worker can start multiple Executor, and each Executor is a JVM process.

Standalone mode

Spark:// . Fault tolerance and support for distributed deployment.

Driver is outside the cluster and can be any client program. Master can be deployed on separate processes, even on separate machines, but only one can be active. Worker is deployed on a separate process, and it is recommended to deploy on a separate machine.

YARN mode

Yarn mode is a mode in which task management and resource scheduling functions are handed over to YARN framework for processing. It is divided into two modes: yarn-client and yarn-cluster.

Yarn-client is suitable for interaction and debugging, and you want to see the output of the application immediately; yarn-cluster is suitable for production environments.

In yarn-cluster mode, driver runs in AM (ApplicationMaster), which is responsible for requesting resources from YARN and monitoring the running status of jobs. After the user has submitted the job, client can be closed and the job will continue to run on YARN.

Yarn-cluster mode is not suitable for running interactive types of jobs. In yarn-client mode, AM (ApplicationMaster) only requests YARN that executor,client will communicate with the requested executor to schedule work, and client cannot leave.

Mesos mode

The operation mode is similar to YARN, which is divided into two modes: client and cluster. Resource schedulers are divided into coarse-grained (default) and fine-grained (not recommended).

Loyal to technology, love sharing. Welcome to the official account: java big data programming to learn more technical content.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.