Shulou(Shulou.com)05/31 Report--

This article mainly introduces the example analysis of bounded data and unbounded data in Flink. The article is very detailed and has a certain reference value. Interested friends must read it!

Bounded data set

Bounded data sets are very familiar to developers, and in regular processing, we will obtain data from Mysql, text, etc., for calculation and analysis. When we deal with this kind of data, the characteristic is that the data is static. In other words, no more additions are made. Or that is to say, the additional write operation is not considered at the time of reprocessing. So bounded data sets, or time boundaries. The results are calculated within a certain period of time. Then this kind of calculation is called batch computing, batch processing. Batch Processing

For example: calculate the current order quantity. Or read the current mysql data into the file.

Unbounded data set

For some scenarios, continuous computing like Kafka is considered to be an unbounded dataset, which is continuously changed and continuously appended. For example: server signaling, network transport stream, real-time log information and so on. The calculation method for this kind of continuously changing and additional data is called stream computing. Streaming Processing

Scene comparison

Unbounded datasets and bounded datasets are a bit similar to ponds and rivers. We only need to calculate all the current fish in the pond when we calculate the number of fish in the pond. So at the moment, how many fish there are in the pond is the result. The unbounded data set is similar to the fish in the river. In the process of running to the sea, the fish will flow through and into the sea all the time. Then calculating the number of fish is like a continuous increase.

Bounded data set and unbounded data set is a relatively vague concept, if the data is processed by a processing engine, then it can be considered unbounded. Then if the calculation is carried out every minute, hour or day, then it is considered that the data of this period of time is relatively bounded. Bounded data can pass the data through the computing engine one by one, resulting in an unbounded data set. Therefore, bounded data sets and unbounded data sets can be interchangeable. Therefore, the industry also began to pursue a unified framework for batch flow.

The frameworks that can implement both batch and stream processing are Apache Spark and Apache Flink, while the flow processing scenario of Apache Spark is a micro-batch scenario, that is, it initiates a calculation at a specific time interval. Not every item triggers the calculation. This is equivalent to dividing an unbounded data set into a small amount of bounded data.

Based on the characteristics of bounded data set and unbounded data set, Apache Flink finally mixes batch processing and stream processing into the same set of engine. Users can use Apache Flink engine to realize batch processing and flow processing tasks at the same time.

The above is all the contents of the article "sample Analysis of bounded and Unbounded data in Flink". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.