Shulou(Shulou.com)06/03 Report--

I believe all of you who read this article, like me, have some doubts about the choice of Hadoop and Apache Spark. After checking a lot of information today, let's talk about the comparison and choice of these two platforms to see which is better for work and development.

1. Hadoop and Spark

1.Spark

Spark is a platform for implementing fast and general-purpose cluster computing. In terms of speed, Spark extends the widely used MapReduce computing model and efficiently supports more computing models, including interactive queries and stream processing.

The Spark project contains several tightly integrated components. The core of Spark is a computing engine that schedules, distributes and monitors applications that are composed of many computing tasks and run on multiple working machines or a computing cluster.

2.Hadoop

Hadoop is a distributed system infrastructure developed by the Apache Foundation. Users can develop distributed programs without knowing the underlying details of the distribution. Make full use of the power of the cluster for high-speed operation and storage. The core design of Hadoop's framework is: HDFS and MapReduce. HDFS provides storage for massive data, while MapReduce provides computing for massive data.

Big data is the direction of future development, challenging our analytical ability and the way we perceive the world. Therefore, we keep pace with the times, meet the changes, and continue to grow. Big data studies the QQ group 606-859-705 to discuss progressive learning together.

II. Differences and similarities

The level of problem solving is different.

First of all, Hadoop and Apache Spark are both big data frameworks, but they exist for different purposes. Hadoop is essentially more of a distributed data infrastructure: it dispatches large data sets to multiple nodes in a cluster of ordinary computers for storage, meaning you don't need to buy and maintain expensive server hardware. At the same time, Hadoop also indexes and tracks the data, making big data's processing and analysis more efficient than ever before. Spark is a tool specially used to deal with those distributed storage big data, it will not carry out distributed data storage.

The two can be combined and separated.

Hadoop not only provides the common HDFS distributed data storage function, but also provides a data processing function called MapReduce. So here we can put aside Spark and use Hadoop's own MapReduce to complete the data processing.

On the contrary, Spark does not have to be attached to Hadoop in order to survive. But as mentioned above, after all, it does not provide a file management system, so it must be integrated with other distributed file systems to work. Here we can choose Hadoop's HDFS or other cloud-based data system platforms. But Spark is still used on Hadoop by default, after all, everyone thinks that their combination is the best.

By the way, what is mapreduce: we need to count all the books in the library. You count bookshelf 1, I count bookshelf 2. This is "Map". The more we are, the faster we will count books. Now let's get together and add up everyone's statistics. This is "Reduce".

Spark data processing speed kills MapReduce in seconds.

Spark is much faster than MapReduce because of the way it handles data. MapReduce processes the data step by step: "read the data from the cluster, process it once, write the results to the cluster, read the updated data from the cluster, do the next processing, write the results to the cluster, and so on." This is how Booz Allen Hamilton data scientist Kirk Borne parses.

Spark, by contrast, does all the data analysis in memory in close to "real time": "read the data from the cluster, do all the necessary analytical processing, write the results back to the cluster, done," Born said. The batch processing speed of Spark is nearly 10 times faster than that of MapReduce, and the data analysis speed in memory is nearly 100 times faster. If the data and result requirements that need to be processed are mostly static, and you have the patience to wait for the batch to complete, MapReduce's approach is perfectly acceptable.

But if you need convection data for analysis, such as data collected by sensors from the factory, or if your application requires multiple data processing, then you may be more likely to use Spark for processing. Most machine learning algorithms require multiple data processing. In addition, the application scenarios that usually use Spark include the following aspects: real-time market activities, online product recommendation, network security analysis, machine diary monitoring and so on.

Recovery recovery

The two disaster recovery methods are very different, but both are very good. Because Hadoop writes each processed data to disk, it is inherently flexible in dealing with system errors. Spark's data objects are stored in a distributed data cluster called resilient distributed dataset (RDD: Resilient Distributed Dataset). "these data objects can be placed in memory or on disk, so RDD can also provide completed disaster recovery functions."

Third, which one should you learn?

In fact, as we know, Spark is indeed a rising star in big data industry. Compared with Hadoop, Spark has many advantages. The reason why Hadoop can be fully recognized in big data industry is mainly because:

Hadoop solves the problem of reliable storage and processing of big data.

The open source of Hadoop, which can make many big data employees find inspiration in it, convenient and practical.

After years of development, Hadoop has a complete ecosystem.

HDFS provides highly reliable file storage on a cluster of ordinary PC, solving the problem of server or hard board failure by saving multiple copies of the block.

MapReduce provides a model through a simple abstraction of Mapper and Reducer, which can concurrently process a large number of data sets on an unreliable cluster composed of dozens to hundreds of PC, while hiding computing details such as concurrency, distribution and fault recovery.

Hadoop also has many limitations and shortcomings, generally speaking, in the case of the continuous expansion of the amount of data, the operation speed of Hadoop will become more and more difficult. Although at this stage, Hadoop still has a high frequency of application in big data industry, but it is not difficult to imagine the dilemma faced by Hadoop when the amount of data increases by several orders of magnitude after a few years. The computing speed of Spark is 1% or even faster than that of Hadoop. Therefore, in the future, Spark will inevitably replace Hadoop and dominate big data's industry.

Is it possible to skip Hadoop and just learn Spark? Of course not, for the following reasons:

At this stage, Hadoop still dominates the field of big data, we can learn advanced technology, but it is more for the current stage of employment, as far as the current stage is concerned, learning big data must learn Hadoop.

There are many classical ideas in MapReduce, which are worth learning, which are very helpful for us to understand big data.

To be exact, Spark replaces MapReduce in Hadoop, not that Hadoop,Hadoop is a toolkit, and Spark, like MapReduce, is just a tool.

Conclusion:

If you are developing into algorithmic engineering in the industry, you need to learn both, Hadoop to understand and Spark to be familiar with. If you are a big data researcher, you should be proficient in both. So, the suggestion here is that for those who are interested in developing in fields such as ML and big data, you can follow the path of Java-Hadoop-Spark. If you have the foundation of C++ and SQL, then the learning curve will not be particularly steep. For spark, it will be more helpful to learn a little Scala.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.