Shulou(Shulou.com)06/01 Report--

This article focuses on "what are the differences between Hadoop and Spark cluster technology". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Next, let the editor take you to learn "what are the differences between Hadoop and Spark cluster technology?"

The level of problem solving is different.

First of all, Hadoop and Apache Spark are both big data frameworks, but they exist for different purposes. Hadoop is essentially more of a distributed data infrastructure: it dispatches large data sets to multiple nodes in a cluster of ordinary computers for storage, meaning you don't need to buy and maintain expensive server hardware.

At the same time, Hadoop also indexes and tracks the data, making big data's processing and analysis more efficient than ever before. Spark is a tool specially used to deal with those distributed storage big data, it will not carry out distributed data storage.

The two can be combined and separated.

Hadoop not only provides the common HDFS distributed data storage function, but also provides a data processing function called MapReduce. So here we can put aside Spark and use Hadoop's own MapReduce to complete the data processing.

On the contrary, Spark does not have to be attached to Hadoop in order to survive. But as mentioned above, after all, it does not provide a file management system, so it must be integrated with other distributed file systems to work. Here we can choose Hadoop's HDFS or other cloud-based data system platforms. But Spark is still used on Hadoop by default, after all, everyone thinks that their combination is the best.

The following is the most concise and clear analysis of MapReduce from online excerpts:

We need to count all the books in the library. You count bookshelf 1, I count bookshelf 2. This is "Map". The more we are, the faster we will count books.

Now let's get together and add up everyone's statistics. This is "Reduce".

Spark data processing speed kills MapReduce in seconds.

Anyone familiar with Hadoop should know that users write a program first, we call it Mapreduce program, a Mapreduce program is a Job, and a Job can have one or more Task,Task and can be divided into Map Task and Reduce Task, as shown in the following figure:

In Spark, there is also the concept of Job, but unlike Job in Mapreduce, Job here is not the highest level of granularity of the job, and there is only the concept of Application on it.

An Application is associated with a SparkContext, and there can be one or more Job in each Application, and the Job can be run in parallel or serially. An Action in Spark can trigger the operation of a Job. Job also contains a number of Stage,Stage is divided by Shuffle. Multiple Task are included in Stage, and multiple Task form Task Set. The relationship between them is shown in the following figure:

Each Task in Mapreduce runs in its own process, and when the Task finishes running, the process ends. Unlike Mapreduce, multiple Task in Spark can run in one process, and the life cycle of this process is the same as Application, even if there is no Job running.

What are the benefits of this model? Can speed up the running speed of Spark! Tasks can start quickly and process data in memory. But the drawback of this model is coarse-grained resource management, where each Application has a fixed number of executor and a fixed amount of memory.

Spark is much faster than MapReduce because of the way it handles data. MapReduce processes the data step by step: "read the data from the cluster, process it once, write the results to the cluster, read the updated data from the cluster, do the next processing, write the results to the cluster, and so on." This is how Booz Allen Hamilton data scientist Kirk Borne parses.

Spark, by contrast, does all the data analysis in memory in close to "real time": "read the data from the cluster, do all the necessary analytical processing, write the results back to the cluster, done," Born said. The batch processing speed of Spark is nearly 10 times faster than that of MapReduce, and the data analysis speed in memory is nearly 100 times faster.

If the data and result requirements that need to be processed are mostly static, and you have the patience to wait for the batch to complete, MapReduce's approach is perfectly acceptable.

But if you need convection data for analysis, such as data collected by sensors from the factory, or if your application requires multiple data processing, then you may be more likely to use Spark for processing.

Most machine learning algorithms require multiple data processing. In addition, the application scenarios that usually use Spark include the following aspects: real-time market activities, online product recommendation, network security analysis, machine diary monitoring and so on.

Disaster recovery

The two disaster recovery methods are very different, but both are very good. Because Hadoop writes each processed data to disk, it is inherently flexible in dealing with system errors.

Spark's data objects are stored in a distributed data cluster called resilient distributed datasets (RDD: Resilient Distributed Dataset). "these data objects can be placed either in memory or on disk, so RDD can also provide completed disaster recovery," Borne pointed out.

At this point, I believe you have a deeper understanding of "what are the differences between Hadoop and Spark cluster technology?" you might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.