Shulou(Shulou.com)05/31 Report--

This article mainly introduces what are the similarities and differences between big data frame Hadoop and Spark, the article is very detailed, has a certain reference value, interested friends must read it!

The level of problem solving is different.

First of all, Hadoop and Apache Spark are both big data frameworks, but they exist for different purposes. Hadoop is essentially more of a distributed data infrastructure: it dispatches large data sets to multiple nodes in a cluster of ordinary computers for storage, meaning you don't need to buy and maintain expensive server hardware.

At the same time, Hadoop also indexes and tracks the data, making big data's processing and analysis more efficient than ever before. Spark is a tool specially used to deal with those distributed storage big data, it will not carry out distributed data storage.

The two can be combined and separated.

Hadoop not only provides the common HDFS distributed data storage function, but also provides a data processing function called MapReduce. So here we can put aside Spark and use Hadoop's own MapReduce to complete the data processing.

On the contrary, Spark does not have to be attached to Hadoop in order to survive. But as mentioned above, after all, it does not provide a file management system, so it must be integrated with other distributed file systems to work. Here we can choose Hadoop's HDFS or other cloud-based data system platforms. But Spark is still used on Hadoop by default, after all, everyone thinks that their combination is the best.

The following is the most concise and clear analysis of MapReduce extracted from the Internet by the Zhuhai rudder of Tiandi Society, in which people can be understood as computers:

We need to count all the books in the library. You count bookshelf 1, I count bookshelf 2. This is Map. The more we are, the faster we will count books.

Now let's get together and add up everyone's statistics. This is Reduce.

Spark data processing speed kills MapReduce in seconds.

Spark is much faster than MapReduce because of the way it handles data. MapReduce processes the data step by step, reads the data from the cluster, processes it once, writes the result to the cluster, reads the updated data from the cluster, carries on the next processing, writes the result to the cluster, and so on, as parsed by Kirk Borne, a data scientist at Booz Allen Hamilton.

Spark, by contrast, completes all data analysis in near real time in memory, reads data from the cluster, completes all necessary analysis processing, writes the results back to the cluster, and is done, Born said. The batch processing speed of Spark is nearly 10 times faster than that of MapReduce, and the data analysis speed in memory is nearly 100 times faster.

If the data and result requirements that need to be processed are mostly static, and you have the patience to wait for the batch to complete, MapReduce's approach is perfectly acceptable.

But if you need convection data for analysis, such as data collected by sensors from the factory, or if your application requires multiple data processing, then you may be more likely to use Spark for processing.

Most machine learning algorithms require multiple data processing. In addition, the application scenarios that usually use Spark include the following aspects: real-time market activities, online product recommendation, network security analysis, machine diary monitoring and so on.

Disaster recovery

The two disaster recovery methods are very different, but both are very good. Because Hadoop writes each processed data to disk, it is inherently flexible in dealing with system errors.

Spark data objects are stored in a distributed data cluster called resilient distributed dataset (RDD: Resilient Distributed Dataset). These data objects can be placed either in memory or on disk, so RDD can also provide completed disaster recovery, Borne noted.

These are all the contents of this article entitled "what are the similarities and differences between big data frame Hadoop and Spark". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.