How to parse the core terms of spark kernel
This article introduces the relevant knowledge of "how to parse the core terms of spark kernel". In the operation process of actual cases, many people will encounter such difficulties. Next, let Xiaobian lead you to learn how to deal with these situations! I hope you can read carefully and learn something!
Application:
Application is the spark user who created the SparkContext instance object and contains the Driver program:
Spark-shell is an application because spark-shell creates a SparkContext object named sc:
Job:
Corresponding to Spark actions, each action such as count, saveAsTextFile, etc. corresponds to a job instance that contains parallel computation of multiple tasks.
Driver Program：
Program that runs main function and creates SparkContext instance
Cluster Manager：
Cluster resource management external services. There are three cluster resource managers on Spark: standalone, yarn, and mesos. The standalone mode of Spark can meet the requirements of most Spark computing environments for cluster resource management. Yarn and mesos are basically considered only when running multiple sets of computing frameworks in a cluster.
Worker Node：
A worker node in a cluster that can run application code, equivalent to a slave node in Hadoop.
Executor：
A worker process started for an application on a Worker Node assigns tasks to run in the process, and is responsible for storing data in memory or disk. It must be noted that each application will only have one Executor on a Worker Node, and the tasks of the application will be processed concurrently through multithreading within the Executor.
State：
A job can be divided into many tasks, each group of tasks is called state, this MapReduce map and reduce tasks are very similar, the basis for dividing the state is that: the state usually starts due to reading external data or shuffle data, the end of a state is usually due to shuffle (such as reduceByKey operation) or the end of the whole job, such as putting data on storage systems such as hdfs.
"Spark kernel core terminology how to parse" the content of the introduction here, thank you for reading. If you want to know more about industry-related knowledge, you can pay attention to the website. Xiaobian will output more high-quality practical articles for everyone!
