Shulou(Shulou.com)06/02 Report--

This article introduces what are the answers to Spark knowledge points, the content is very detailed, interested friends can refer to, hope to be helpful to you.

Spark is a fast and universal cluster computing platform. It is a memory-based iterative computing framework, which is suitable for applications that need to manipulate specific data sets multiple times. The knowledge points that often need to use Spark,Spark in the large database architecture has always been the focus of big data's interview questions. The following collates a summary of Spark knowledge points.

1. What is the core of Spark?

RDD is the basic abstraction of Spark and the abstract use of distributed memory. It implements the abstract implementation of manipulating distributed data sets by manipulating local sets. RDD is also the core of Spark. It represents a set of data that has been partitioned, immutable and can be operated in parallel. Different dataset formats correspond to different RDD implementations.

RDD must be serializable. RDD can be cache into memory, and the results of each operation on the RDD dataset can be stored in memory, and the next operation can be entered directly from memory, saving a lot of disk IO operations of MapReduce. For the iterative operation of the more common machine learning algorithms, interactive data mining, the efficiency is greatly improved.

2. What are the applicable scenarios for Spark?

Due to the nature of RDD, Spark is not suitable for applications with asynchronous fine-grained status updates, such as storage of web services or incremental web crawlers and indexes. It is not suitable for the application model of incremental modification. Generally speaking, Spark has a wide range of applications and is more general.

3. What are the programming languages supported by Spark?

Spark exposes RDD operations through integration with programming languages, similar to DryadLINQ and FlumeJava. Each dataset is represented as a RDD object, and the operation on the dataset is represented as an operation on the RDD object.

The main programming languages supported by Spark are Scala, java and python.

Scala

Spark is developed using Scala and defaults to Scala as the programming language. Writing Spark programs is much easier than writing Hadoop MapReduce programs. SparK provides Spark-Shell, which can be tested in Spark-Shell.

Java

Spark supports Java programming, but for using Java, there is no convenient tool like Spark-Shell, other programming is the same as Scala programming, because they are all languages on JVM, Scala and Java can be interoperable, Java programming interface is actually the encapsulation of Scala.

Python

Now Spark also provides Python programming interface, Spark uses py4j to achieve the interoperation between python and java, thus realizing the use of python to write Spark programs. Spark also provides pyspark, a python shell for Spark, which allows you to write Spark programs in Python interactively.

What are the answers about Spark knowledge points to share here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.