Shulou(Shulou.com)05/31 Report--

This article mainly explains "what is the operation mechanism of Spark Streaming". The explanation in this article is simple and clear, easy to learn and understand. Please follow the ideas of Xiaobian and go deep into it slowly to study and learn "what is the operation mechanism of Spark Streaming" together.

All data that cannot be processed in real-time stream is invalid data. In the era of stream processing, SparkStreaming has strong appeal and broad development prospects. Coupled with the ecosystem of Spark, Streaming can easily call other powerful frameworks such as SQL and MLlib.

The Spark Streaming runtime is not so much a streaming framework on Spark Core as one of the most complex applications on Spark Core. If you can master the complex application of Spark streaming, then other complex applications will be no problem. The choice of Spark Streaming as the entry point for version customization is also the trend.

For Spark Streaming, when data streams from different data sources come in, they form a fixed set of data sets or events (e.g. from flumes and kafka) based on fixed time intervals. This coincides with RDD being based on a fixed data set; in fact, the RDD Graph processed by DStream over a fixed time interval is based on a batch data set.

As can be seen from the above figure, on each batch, the RDD dependencies of the spatial dimension are the same. The difference is that the data size and content of the five batches are different, so it is said that the generated RDD dependencies are different instances. Therefore, the Graph of RDD is derived from the Graph of DStream, that is to say, DStream is the template of RDD. Different time intervals generate different instances of RDD Graph.

Starting with Spark Streaming itself:

1. RDD DAG generation template required: DStream Graph

2 Requires Timeline-based job controllers

3 requires inputStreamings and outputStreamings to represent data inputs and outputs

4 The specific job runs on top of Spark Cluster. Since streaming can be digested regardless of whether the cluster can be digested, system fault tolerance is crucial at this time.

5 Transaction processing, we want incoming data to be processed, and only once. How to guarantee Exactly once transaction semantics in case of processing crash.

Interpreting DStream from Source Code

From here, we can see that DStream is the core of Spark Streaming, just like the core of Spark Core is RDD, which also has dependency and compute. More importantly, the following code:

This is a HashMap with time as key and RDD as value, which also proves that over time, RDD is continuously generated, jobs with dependencies are generated, and run on the cluster through jobScheduler. Once again, DStream is the template for RDD.

DStream can be said to be logical level, RDD is physical level, and what DStream expresses is finally realized through the transformation of RDD. The former is a higher-level abstraction and the latter is a low-level implementation. DStream is actually the encapsulation of RDD set in time dimension. The relationship between DStream and RDD is that RDD is continuously generated with the passage of time. The operation on DStream is to operate RDD in fixed time.

Thank you for reading, the above is "Spark Streaming operation mechanism is what" content, after the study of this article, I believe we have a deeper understanding of Spark Streaming operation mechanism is what this problem, the specific use of the situation also needs to be verified by practice. Here is, Xiaobian will push more articles related to knowledge points for everyone, welcome to pay attention!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.