Shulou(Shulou.com)06/02 Report--

This article mainly introduces what apache spark is, the article is very detailed, has a certain reference value, interested friends must read it!

Spark is an open source cluster computing system based on memory computing, which aims to make data analysis faster. Spark is very small and developed by a small team based on Matei at the AMP Lab at the University of California, Berkeley. The language used is Scala, and the code in the core part of the project has only 63 Scala files, which is very short and concise.

Spark is an open source cluster computing environment similar to Hadoop, but there are some differences between the two. These useful differences make Spark superior in some workloads. In other words, Spark enables in-memory distributed datasets to optimize iterative workloads in addition to interactive queries.

Spark is implemented in the Scala language and uses Scala as its application framework. Unlike Hadoop, Spark and Scala can be tightly integrated, where Scala can manipulate distributed datasets as easily as local collection objects.

Although Spark was created to support iterative jobs on distributed datasets, it is actually a complement to Hadoop and can be run in parallel in the Hadoop file system. This behavior can be supported through a third-party cluster framework called Mesos. Developed by AMP Lab (Algorithms, Machines, and People Lab) at the University of California, Berkeley, Spark can be used to build large, low-latency data analysis applications.

Spark Cluster Computing Architecture

Although Spark has similarities with Hadoop, it provides a new cluster computing framework with useful differences. First, Spark is designed for specific types of workloads in cluster computing, that is, workloads that reuse working data sets (such as machine learning algorithms) between parallel operations. To optimize these types of workloads, Spark introduces the concept of memory cluster computing, which caches data sets in memory in memory cluster computing to reduce access latency.

Spark also introduces an abstraction called resilient distributed datasets (RDD). A RDD is a collection of read-only objects distributed over a set of nodes. These collections are resilient and can be rebuilt if part of the dataset is missing. The process of rebuilding a partial dataset depends on a fault-tolerant mechanism that maintains "lineage" (that is, information that allows partial datasets to be reconstructed based on the data derivation process). RDD is represented as a Scala object and can be created from a file; a parallelized slice (across nodes); a transformed form of another RDD; and eventually completely change the persistence of the existing RDD, such as requests cached in memory.

Applications in Spark are called drivers, which implement operations performed on a single node or in parallel on a set of nodes. Similar to Hadoop, Spark supports single-node clusters or multi-node clusters. For multi-node operations, Spark relies on the Mesos cluster manager. Mesos provides an effective platform for resource sharing and isolation of distributed applications. This setting allows Spark and Hadoop to coexist in a shared pool of nodes.

The above is all the content of what apache spark is, thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.