Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "what is the difference between Spark and Hadoop big data computing framework". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

What is the difference between Spark and Hadoop big data computing framework? ApacheSpark is a fast and general computing engine designed for large-scale data processing, while Hadoop is a distributed system infrastructure developed by the Apache Foundation. Hadoop and Spark are both big data frameworks, but they exist for different purposes. Hadoop is essentially a distributed data infrastructure, which distributes huge data sets to multiple nodes in a cluster composed of ordinary computers for storage, as well as computing and processing functions. Spark is a tool specially used to deal with those distributed storage of big data and will not store distributed data.

What is Spark?

Spark is a general big data computing framework, just like the traditional big data technology Hadoop MapReduce, Hive engine, and Storm streaming real-time computing engine. Spark includes various computing frameworks common in big data's field, such as SparkCore for offline computing, SparkSQL for interactive query, SparkStreaming for real-time streaming computing, SparkMLlib for machine learning, and SparkGraphX for graph computing.

Spark is mainly used for big data's computing, while Hadoop is mainly used for big data's storage (such as HDFS, Hive, HBase, etc.) and resource scheduling (Yarn).

What is Hadoop?

Hadoop is the general name of the project, which is composed of HDFS and MapReduce. HDFS is an open source implementation of GoogleFileSystem (GFS). MapReduce is an open source implementation of GoogleMapReduce. ApacheHadoop software library is a framework that allows large data sets to be processed across computer clusters using a simple programming model. It is originally designed to expand a single server into a cluster of thousands of machines to provide computing services for big data, each of which provides local computing and storage services.

Hadoop and Spark are both big data computing frameworks, but each has its own advantages. The differences between Spark and Hadoop are as follows:

1. Programming method

When calculating data in MapReduce of Hadoop, the calculation process must be transformed into two processes: Map and Reduce, so it is difficult to describe the complex data processing process, while the calculation model of Spark is not limited to Map and Reduce operations, but also provides a variety of operation types of data sets, and the programming model is more flexible than MapReduce.

2. Data storage

When Hadoop's MapReduce calculates, the intermediate results generated each time are stored on the local disk, while the intermediate results generated by Spark are stored in memory.

3. Data processing

Every time Hadoop performs data processing, it needs to load data from the disk, which results in a large cost of disk Imax O, while Spark only needs to load the data into memory, and then directly loads the intermediate result data set in memory, which reduces the disk overhead of 10.

4. Data fault tolerance

The intermediate result data of MapReduce computing is saved in disk, and the backup mechanism is implemented at the bottom of Hadoop framework, which ensures data fault tolerance; similarly, SparkRDD implements the fault tolerance mechanism based on Lineage and the fault tolerance mechanism of setting checkpoints, which makes up for the problem of power outage and loss when data is processed in memory. In the performance comparison between Spark and Hadoop, the obvious defect is the high latency of MapReduce computing in Hadoop, which can not meet the requirements of real-time and fast computing required by the explosive data growth.

When using HadoopMapReduce for calculation, the intermediate results produced by each calculation need to be read and written from the disk, which greatly increases the cost of the disk, while when using Spark for calculation, the data in the disk needs to be read into memory first, and the resulting data is no longer written to the disk, but is iterated directly in memory, thus avoiding the unnecessary overhead caused by frequently reading data from the disk.

Spark is an open source cluster computing environment similar to Hadoop, but the difference makes Spark more superior in some workloads. Spark enables in-memory distributed data sets, which can optimize iterative workloads in addition to interactive queries. Spark is implemented in the Scala language and uses Scala as its application framework. Unlike Hadoop, Spark and Scala can be tightly integrated, where Scala can manipulate distributed datasets as easily as local collection objects.

This is the end of the content of "what is the difference between Spark and Hadoop big data computing framework". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.