Shulou(Shulou.com)06/01 Report--

This article mainly introduces "what's the difference between park and Flink". In daily operation, I believe many people have doubts about the differences between park and Flink. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful to answer the questions of "what's the difference between park and Flink?" Next, please follow the editor to study!

Introduction to Spark

Spark is a memory-based computing framework, the computing speed is very fast. If you want to dock with external data, such as HDFS reading data, you need to build a Hadoop cluster in advance. Apache Spark is an open source cluster computing framework. MapReduce relative to Hadoop will store intermediary data to disk after running. Spark uses in-memory computing technology to analyze operations in memory when the data has not been written to the hard disk.

Introduction to Flink

Flink is a distributed processing engine for streaming data and batch data. It is mainly implemented by Java code. At present, it mainly depends on the contribution of the open source community. For Flink, the main scenario it has to deal with is stream data, and batch data is only a limited special case of stream data. In other words, Flink treats all tasks as streams, which is its biggest feature. Flink can support local fast iterations, as well as some circular iterative tasks.

Comparison between Flink and Spark

Both Spark and Flink support batch and streaming processing, so let's compare these two popular data processing frameworks in various aspects. First of all, the two data processing frameworks have a lot in common.

Are based on memory computing.

Both have unified batch and streaming APl, and both support programming interfaces similar to SQL

Both support many of the same conversion operations, and are programmed in a functional programming mode similar to Scala Collection APl

All have a perfect error recovery mechanism.

Both support semantic consistency of Exactly once.

Of course, the differences between them are also quite obvious, and we can look at them from four different angles.

From the point of view of stream processing, Spark is based on micro-batch processing, which treats the stream data as small batch data blocks separately, so the delay can only achieve seconds. On the other hand, Flink is based on the processing of each event and will be processed immediately whenever there is new data input. it is a real streaming computing and supports millisecond computing. For the same reason, Spark only supports time-based window operations (processing time or event time), while Flink supports window operations that are very flexible, supporting not only time windows, but also windows based on data itself. Developers are free to define the window operations they want.

From the perspective of SQL function, Spark and Flink provide SparkSQL and Table APl interactive support for SQL, respectively. Compared with the two, Spark supports SQL better, and the corresponding optimization, expansion and performance are better, while Flink still has a lot of room for improvement in SQL support.

From the perspective of iterative computation, Spark supports machine learning very well, because the intermediate calculation results can be cached in memory to speed up the operation of machine learning algorithms. But most machine learning algorithms are actually a cyclic data stream, which is represented by acyclic graph in Spark. On the other hand, Flink supports cyclic data flow at run time, so that machine learning algorithms can be operated more effectively.

From the perspective of the corresponding ecosystem, the community of Spark is undoubtedly more active. Spark can be said to have the largest number of open source contributors under Apache, and there are many different libraries to use in different scenarios. Because Flink is relatively new, the open source community at this stage is not as active as Spark, and the functions of various libraries are not as comprehensive as Spark. But Flink is still developing, and various functions are gradually improving.

How to choose Spark and Flink

For the following scenarios, you can choose Spark.

Batch data processing with a very large amount of data and complex logic, and has high requirements for computational efficiency (for example, using big data analysis to build a recommendation system for personalized recommendation, advertising, etc.)

Interactive query based on historical data requires fast response.

The delay of data processing based on real-time data stream is between hundreds of milliseconds and several seconds.

At this point, the study on "what's the difference between park and Flink" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.