Shulou(Shulou.com)06/01 Report--

This article mainly introduces "what is the relationship between Spark and Spark-Streaming". In daily operation, I believe many people have doubts about the relationship between Spark and Spark-Streaming. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful to answer the doubts about "what is the relationship between Spark and Spark-Streaming?" Next, please follow the editor to study!

The spark program uses a spark application instance to process a batch of historical data at one time. Spark streaming converts the continuously input data flow into multiple batch fragments, and uses a batch of spark application examples for processing, focusing on Steaming. What we often say that Spark-Streaming relies on Spark Core means that the core framework of actual computing is still spark. We are still the last clich é official picture:

In principle, what do we need to transform spark-streaming into a traditional spark?

There are four things you need to build:

A static RDD DAG template to represent the processing logic

A dynamic working controller that splits continuous streaming data into pieces of data and copies a new RDD according to the template

An example of DAG to process data fragments

Receiver generates and imports the original data; Receiver merges the received data into data blocks and stores them in memory or hard disk for subsequent batch RDD consumption; the guarantee of long-running tasks, including reconstruction after the failure of input data, and reset after the failure of processing tasks.

At this point, the study on "what is the relationship between Spark and Spark-Streaming" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.