Shulou(Shulou.com)06/02 Report--

This article mainly introduces the relevant knowledge of "what are the problems related to Spark". The editor shows you the operation process through actual cases, and the operation method is simple, fast and practical. I hope this article "what are the problems related to Spark" can help you solve the problem.

1. What is the core of Spark?

RDD is the basic abstraction of Spark and the abstract use of distributed memory. It implements the abstract implementation of manipulating distributed data sets by manipulating local sets. RDD is also the core of Spark. It represents a set of data that has been partitioned, immutable and can be operated in parallel. Different dataset formats correspond to different RDD implementations.

RDD must be serializable. RDD can be cache into memory, and the results of each operation on the RDD dataset can be stored in memory, and the next operation can be entered directly from memory, saving a lot of disk IO operations of MapReduce. For the iterative operation of the more common machine learning algorithms, interactive data mining, the efficiency is greatly improved.

2. What are the applicable scenarios for Spark?

Due to the nature of RDD, Spark is not suitable for applications with asynchronous fine-grained status updates, such as storage of web services or incremental web crawlers and indexes. It is not suitable for the application model of incremental modification. Generally speaking, Spark has a wide range of applications and is more general.

3. What are the programming languages supported by Spark?

Spark exposes RDD operations through integration with programming languages, similar to DryadLINQ and FlumeJava. Each dataset is represented as a RDD object, and the operation on the dataset is represented as an operation on the RDD object. The main programming languages supported by Spark are Scala, java and python.

Scala

Spark is developed using Scala and defaults to Scala as the programming language. Writing Spark programs is much easier than writing Hadoop MapReduce programs. SparK provides Spark-Shell, which can be tested in Spark-Shell.

Java

Spark supports Java programming, but for using Java, there is no convenient tool like Spark-Shell, other programming is the same as Scala programming, because they are all languages on JVM, Scala and Java can be interoperable, Java programming interface is actually the encapsulation of Scala.

Python

Now Spark also provides Python programming interface, Spark uses py4j to achieve the interoperation between python and java, thus realizing the use of python to write Spark programs. Spark also provides pyspark, a python shell for Spark, which allows you to write Spark programs in Python interactively.

This is the end of the content about "what are the issues related to Spark"? thank you for your reading. If you want to know more about the industry, you can follow the industry information channel. The editor will update different knowledge points for you every day.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.