Network Security Internet Technology Development Database Servers Mobile Phone Android Software Apple Software Computer Software News IT Information

In addition to Weibo, there is also WeChat

Please pay attention

WeChat public account

Shulou

What are the types of RDD dependencies in Spark

2025-02-25 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Servers >

Share

Shulou(Shulou.com)05/31 Report--

This article introduces the relevant knowledge of "what kinds of RDD dependencies are divided into in Spark". In the operation process of actual cases, many people will encounter such difficulties. Next, let Xiaobian lead you to learn how to deal with these situations! I hope you can read carefully and learn something!

Spark programming model based on RDD

Everything in Spark is based on RDD:

What is RDD? The official explanation was:

This means that every RDD has at least three functions:

Spark comes with a lot of RDD:

There are two main types of RDD:

where transformations are lazy execution, specific actions are required to trigger, and each action operation is a separate job;

Let's look at a concrete example of RDD operations:

There are two special RDD:

These two RDDs are unique in that they are both RDDs with control operations:

Second, RDD dependencies and runtime

In Spark, RDD has dependencies, and dependencies are divided into two types:

One of the benefits of the "Narrow" dependency is the ability to perform internal pipeline operations:

When running, it runs in parallel with RDD as a unified abstract:

A further detailed RDD parallelization process is shown below:

Like all distributed structures, Spark distributed clusters are master-slave:

The Spark runtime components are as follows:

The event flow at runtime for Spark is as follows:

"Spark RDD dependencies are divided into several types" of content introduced here, thank you for reading. If you want to know more about industry-related knowledge, you can pay attention to the website. Xiaobian will output more high-quality practical articles for everyone!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.

Views: 0

*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.

Share To

Servers

Wechat

© 2024 shulou.com SLNews company. All rights reserved.

12
Report