Shulou(Shulou.com)06/01 Report--

This article mainly explains "what is the difference between hadoop and spark". The explanation in this article is simple and clear, easy to learn and understand. Please follow the ideas of Xiaobian and go deep into it slowly to study and learn "what is the difference between hadoop and spark" together!

Hadoop and spark are analyzed in four aspects:

Purpose: First of all, it needs to be clear that both hadoophe spark are big data frameworks, even though the purpose of their existence is different. Hadoop is a distributed data infrastructure that distributes large data sets to multiple nodes in a cluster of several computers for storage. Spark is a tool specifically designed to process large amounts of distributed data, and Spark itself does not store distributed data.

Deployment of both: The core design of Hadoop's framework is HDFS and MapReduce. HDFS provides storage for massive amounts of data, and MapReduce provides computation for massive amounts of data. So using Hadoop, you can put aside spark and directly use Hadoop's own mapreduce to complete the data processing. Spark does not provide a file management system, but it is not only attached to Hadoop, it can also choose other cloud-based data system platforms, but the default choice of Spark is Hadoop.

3. Data processing speed: Spark has the advantages of Hadoop and MapReduce, which are better suitable for data mining and machine learning. However, unlike MapReduce, the intermediate output results of Job can be stored in memory, so that it is no longer necessary to read and write HDFS.

Spark is an open source clustered computing environment similar to Hadoop, but there are some useful differences that make Spark superior for certain workloads, in other words, Spark enables in-memory distributed datasets that optimize iterative workloads in addition to providing interactive queries.

4, data security recovery: Hadoop after each processing of data is written to disk, so it is inherently flexible to deal with system errors;spark data objects stored in the data cluster called elastic distributed data set, these data objects can be placed in memory, can also be placed on disk, so spark can also complete the security recovery of data.

Thank you for reading, the above is "what is the difference between hadoop and spark" content, after the study of this article, I believe that we have a deeper understanding of the difference between hadoop and spark, the specific use of the situation also needs to be verified by practice. Here is, Xiaobian will push more articles related to knowledge points for everyone, welcome to pay attention!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.