Shulou(Shulou.com)06/01 Report--

This article mainly explains "what are the differences between Hadoop cluster technology and Spark cluster technology". The content of the explanation in this article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "what are the differences between Hadoop cluster technology and Spark cluster technology"?

Hadoop: distributed batch computing, emphasizing batch processing, often used for data mining and analysis.

Spark: an open source cluster computing system based on memory computing, designed to make data analysis faster. Spark is an open source cluster computing environment similar to Hadoop, but there are some differences between the two. These useful differences make Spark superior in some workloads. In other words, Spark enables in-memory distributed data sets, in addition to providing interactive queries. It can also optimize iterative workloads.

Spark is implemented in the Scala language and uses Scala as its application framework. Unlike Hadoop, Spark and Scala can be tightly integrated, where Scala can manipulate distributed datasets as easily as local collection objects.

Although Spark was created to support iterative jobs on distributed datasets, it is actually a complement to Hadoop and can be run in parallel in the Hadoop file system. This behavior can be supported through a third-party cluster framework called Mesos. Developed by AMP Laboratories (Algorithms,Machines,and People Lab) at the University of California, Berkeley, Spark can be used to build large, low-latency data analysis applications.

Although Spark has similarities with Hadoop, it provides a new cluster computing framework with useful differences. First, Spark is designed for specific types of workloads in cluster computing, that is, workloads that reuse working data sets (such as machine learning algorithms) between parallel operations. In order to optimize these types of workloads, Spark introduces the concept of memory cluster computing, which can cache data sets in memory in memory cluster computing to shorten access latency.

In big data processing, I believe everyone is already familiar with hadoop. Hadoop based on GoogleMap/Reduce provides developers with map and reduce primitives, which makes parallel batch programs very simple and beautiful. Spark provides many types of dataset operations, unlike Hadoop, which only provides Map and Reduce operations. For example, map,filter, flatMap,sample, groupByKey, reduceByKey, union,join, cogroup,mapValues, sort,partionBy and other types of operations, they call these operations Transformations. At the same time, it also provides Count,collect, reduce, lookup, save and other actions. These various types of dataset operations provide convenience for upper-level users. The communication model between processing nodes is no longer the only Data Shuffle mode like Hadoop. Users can name, materialize, control the partition of intermediate results, and so on. It can be said that the programming model is more flexible than Hadoop.

Hadoop and Spark are both big data frameworks and both provide tools for performing common big data tasks. But to be exact, they do not perform the same tasks and are not mutually exclusive. Although Spark is said to be 100 times faster than Hadoop under certain circumstances, it does not have a distributed storage system of its own. Distributed storage is the basis of many big data projects today. It can store PB-level data sets on an almost unlimited number of ordinary computer hard drives, and provides good scalability, as long as the data set increases. Therefore, Spark needs a third-party distributed storage. It is for this reason that many big data projects install Spark on Hadoop. This allows Spark's advanced analytics application to use the data stored in HDFS.

The real advantage of Spark over Hadoop is speed. Most of Spark's operations are in memory, while Hadoop's MapReduce system writes all data back to the physical storage media after each operation. This is to ensure full recovery in the event of a problem, but this is also possible with Spark's flexible distributed data storage.

In addition, Spark outperforms Hadoop in advanced data processing, such as real-time streaming and machine learning. In Bernard's view, this, along with its speed advantage, is the real reason why Spark is becoming more and more popular. Real-time processing means that the data can be submitted to an analytical application as soon as it is captured and get feedback immediately. In a variety of big data applications, this processing has more and more uses, such as recommendation engines used by retailers and industrial machinery performance monitoring in manufacturing. The speed and stream data processing ability of the Spark platform are also very suitable for machine learning algorithms. Such algorithms can learn and improve themselves until they find an ideal solution to the problem. This technology is at the heart of state-of-the-art manufacturing systems (such as predicting when parts will be damaged) and driverless cars. Spark has its own machine learning library MLib, while Hadoop systems need the help of third-party machine learning libraries, such as Apache Mahout.

In fact, although there is some functional overlap between Spark and Hadoop, they are not commercial products and there is no real competition, and companies that make a profit by providing technical support for such free systems tend to provide both services at the same time. For example, Cloudera provides both Spark and Hadoop services and provides the most appropriate advice according to the customer's needs.

Thank you for your reading. The above is the content of "what is the difference between Hadoop cluster technology and Spark cluster technology". After the study of this article, I believe you have a deeper understanding of the difference between Hadoop cluster technology and Spark cluster technology. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.