Shulou(Shulou.com)06/03 Report--

Hadoop has been learning for a long time, it seems that in February and March when a friend gave a domestic Hadoop distribution download address, because it is still in the learning stage to download a three-node learning version to play. In the research, learning hadoop friends can look for a look (release version of the big fast DKhadoop, go to the big fast website should be able to download to.)

When learning hadoop query some information often see a comparison between hadoop and spark, for beginners will inevitably be a little confused what the big difference between the two. I remember when I first came into contact with big data, I also looked up some information on this issue. In this description document of FreeRCH Big Data Integration Development Framework, there is a simple explanation of the difference between Hadoop and spark, but I think the explanation is not particularly detailed. I would like to share with you one of my better explanations:

It compares Hadoop and spark in four main ways:

Purpose: First of all, it needs to be clear that both hadoophe spark are big data frameworks, even though the purpose of their existence is different. Hadoop is a distributed data infrastructure that distributes large data sets to multiple nodes in a cluster of several computers for storage. Spark is a tool specifically designed to process large amounts of distributed data, and Spark itself does not store distributed data.

Deployment of both: The core design of Hadoop framework is HDFS and MapReduce. HDFS provides storage for massive amounts of data, and MapReduce provides computation for massive amounts of data. So using Hadoop, you can put aside spark and directly use Hadoop's own mapreduce to complete the data processing. Spark does not provide a file management system, but it is not only attached to Hadoop, it can also choose other cloud-based data system platforms, but the default choice of Spark is Hadoop.

3. Data processing speed: Spark has the advantages of Hadoop and MapReduce, which can be better applied to data mining and machine learning. However, unlike MapReduce, the intermediate output results of Job can be stored in memory, so that it is no longer necessary to read and write HDFS.

Spark is an open source clustered computing environment similar to Hadoop, but there are some useful differences that make Spark superior for certain workloads, in other words, Spark enables in-memory distributed datasets that optimize iterative workloads in addition to providing interactive queries.

4, data security recovery: Hadoop after each processing of data is written to disk, so it is inherently flexible to deal with system errors;spark data objects stored in the data cluster called elastic distributed data set, these data objects can be placed in memory, can also be placed on disk, so spark can also complete the security recovery of data.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.