Network Security Internet Technology Development Database Servers Mobile Phone Android Software Apple Software Computer Software News IT Information

In addition to Weibo, there is also WeChat

Please pay attention

WeChat public account

Shulou

What is Spark?

2025-04-05 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Servers >

Share

Shulou(Shulou.com)05/31 Report--

This article introduces the knowledge of "what is Spark". In the operation of actual cases, many people will encounter such a dilemma. Then let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

First, what exactly is Spark?

Spark is a general-purpose big data computing platform. Based on the concept of "One Stack to rule them all", it has successfully become an integrated and diversified big data processing platform, easily dealing with real-time streaming computing, SQL interactive query, machine learning and graph computing in big data processing.

Spark is derived from BDAS:

Based on this technology stack, Spark has become a general computing platform for big data:

Second, why is the speed of Spark so fast?

First, let's take a look at the classic process of Hadoop:

MapReduce reads data from disk every time it executes, and stores the data on disk after calculation:

Spark is memory-based:

On the other hand, DAG is also an extremely important reason for the speed of Spark. Here is an example of a DAG diagram:

You can also take a look at a picture on the Internet that describes DAG in more detail:

Based on DAG,Spark, we have a very exquisite job scheduling system:

Dependencies in DAG can be divided into wide and narrow dependencies:

In the DAG diagram, you can optimize operations such as pipeline based on dependencies:

Based on RDD and DAG, parallel computing of the whole Job:

That's all for "what Spark is". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.

Views: 0

*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.

Share To

Servers

Wechat

© 2024 shulou.com SLNews company. All rights reserved.

12
Report