Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to analyze the operation mode of Spark. The content of the article is of high quality, so the editor will share it with you for reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

Two important roles of Spark Operation Mode 1 Spark

1.1 introduction

Spark is a memory-based express, universal, scalable big data analysis engine.

1.2 spark built-in module:

1.3 two important roles of Spark:

Driver (driver): responsible for managing scheduling tasks

Executor (executor): responsible for performing specific tasks

2 Spark operation mode

Local mode: runs on a machine, usually in a hands-on or test environment.

Standalone: build a Mster+Slaves-based resource scheduling cluster, and Spark tasks are submitted to Master to run. It is a scheduling system of Spark itself and does not rely on other frameworks such as Yarn.

Yarn: the Spark client connects directly to Yarn without the need to build an additional Spark cluster. There are two modes of yarn-client and yarn-cluster, and the main difference lies in the running node of the Driver program.

The yarn-client:Driver program runs on the client side and is suitable for interaction and debugging. I want to see the output of app immediately.

The yarn-cluster:Driver program runs on AP (APPMaster), which is started by RM (ResourceManager), and is suitable for production environments.

Mesos: the domestic environment is seldom used.

Comparison of several modes:

3 Local mode (WordCount under)

3.1Basic ideas of Spark-Wordcount

File load load: load fil

Flattening: to divide a sentence into words.

Grouping: put the same words into a group

Aggregation: count the number of members in each group

(the above process is the simplified process of the left picture, which is not very accurate. Follow the steps below to remember it. The right picture is the accurate process.)

3.2 Wordcount code implementation

Description of related methods:

TextFile: used to load fil

FlatMap: for flattening, word segmentation

Map: map every word to meta-ancestor

ReduceByKey: group aggregation according to key

3.3 Wordcount process diagram

4 Yarn mode (key)

4.1 Overview

The Spark client connects directly to Yarn without the need to build an additional Spark cluster. There are two modes of yarn-client and yarn-cluster, and the main difference lies in the running node of the Driver program.

The yarn-client:Driver program runs on the client side and is suitable for interaction and debugging. You want to see the output of app immediately.

The yarn-cluster:Driver program runs on AP (APPMaster), which is started by RM (ResourceManager), and is suitable for production environments.

4.2 Yarn operation mode (key)

Know clearly the process of each step.

Look at the two pictures together.

5 Standalone mode (standalone deployment mode)

Just use spark's own stuff, not other frameworks such as Yarn.

5.1 Overview

Build a Spark cluster made up of Master+Slave, in which Spark runs.

5.2 Standalone mode

Master is equivalent to RM in Yarn.

Worker is equivalent to NM in Yarn.

On how to analyze the Spark operation mode to share here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.