Shulou(Shulou.com)05/31 Report--

This article introduces the knowledge of "what is Spark". In the operation of actual cases, many people will encounter such a dilemma. Then let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

First, what exactly is Spark?

Spark is a general-purpose big data computing platform. Based on the concept of "One Stack to rule them all", it has successfully become an integrated and diversified big data processing platform, easily dealing with real-time streaming computing, SQL interactive query, machine learning and graph computing in big data processing.

Spark is derived from BDAS:

Based on this technology stack, Spark has become a general computing platform for big data:

Second, why is the speed of Spark so fast?

First, let's take a look at the classic process of Hadoop:

MapReduce reads data from disk every time it executes, and stores the data on disk after calculation:

Spark is memory-based:

On the other hand, DAG is also an extremely important reason for the speed of Spark. Here is an example of a DAG diagram:

You can also take a look at a picture on the Internet that describes DAG in more detail:

Based on DAG,Spark, we have a very exquisite job scheduling system:

Dependencies in DAG can be divided into wide and narrow dependencies:

In the DAG diagram, you can optimize operations such as pipeline based on dependencies:

Based on RDD and DAG, parallel computing of the whole Job:

That's all for "what Spark is". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.