Shulou(Shulou.com)06/03 Report--

Many beginners of Hadoop development can't tell what exactly is the connection between Hadoop and Spark?

It is unclear whether Hadoop and Spark are two separate frameworks, or do they have to be interdependent to get the job done?

Today, I will analyze the differences between Hadoop and Spark.

What are Hadoop and Spark respectively?

Hadoop

Hadoop is a distributed system infrastructure. Users can develop distributed programs without knowing the underlying details of the distribution.

Use the power of the cluster for high-speed operation and storage. The core design of Hadoop's framework is: HDFS and MapReduce. HDFS provides storage for massive data, and MapReduce provides computing for massive data.

Spark

Spark is a fast and general computing engine specially designed for large-scale data processing. It is a general parallel framework similar to Hadoop MapReduce and has the advantages of Hadoop MapReduce.

The similarities and differences between Hadoop and Spark can be roughly divided into the following points

1. Different levels of dealing with the problem

Hadoop

Hadoop is essentially more of a distributed data infrastructure: it distributes huge data sets to multiple nodes in a cluster of ordinary computers for storage, and at the same time indexes and tracks the data, greatly improving big data's processing and analysis efficiency.

Spark

Spark, is a tool specially used to deal with those distributed storage big data, and will not store the data on its own.

two。 It can work together or run independently.

Hadoop

Hadoop can store and process data independently, because it not only provides HDFS distributed data storage function, but also provides MapReduce data processing function.

Spark

Spark does not provide a file management system, which must be integrated with other distributed file systems to work. You can choose Hadoop's HDFS, or you can choose other platforms.

3.Spark data processing speed is much faster than MapReduce.

Hadoop

Hadoop is a disk-level calculation, which needs to read data from the disk; it uses the logic of MapReduce to slice the data to deal with a large number of offline data.

Spark

Spark, which completes all data analysis in memory in close to "real-time" time. The batch processing speed of Spark is nearly 10 times faster than that of MapReduce, and the data analysis speed in memory is nearly 100 times faster.

For example, real-time marketing activities, online product recommendations and other scenarios that need streaming data for analysis should use Spark.

4. Disaster recovery

Hadoop

Hadoop writes the processed data to disk, which has an inherent advantage in dealing with system errors.

Spark

Spark's data objects are stored in elastic distributed datasets (RDD:). "these data objects can be placed either in memory or on disk, so RDD also provides complete disaster recovery capabilities.

How to learn Hadoop development in 4 months and find a job with an annual salary of 250000?

Free to share a set of 17 years of the latest Hadoop big data tutorials and 100 Hadoop big data must meet questions.

Because links are often harmonious, friends who need them please add Wechat ganshiyun666 to get the latest download link, marked "51CTO"

The tutorials have helped 300 + people successfully transform Hadoop development, with a starting salary of more than 20K, double the previous salary.

Recorded by Baidu Hadoop core architect (T7 level) personally.

The content includes three parts: basic introduction, Hadoop ecosystem and real business project. Among them, business cases allow you to come into contact with the real production environment and train your development skills.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.