Shulou(Shulou.com)05/31 Report--

Editor to share with you why flink has become the most popular computing engine, I believe that most people do not understand, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to understand it!

First acquaintance of Flink

I learned about Flink at a meeting with the company two years ago, which is a company that does big data security, using big data to analyze security threat warning. At that time, they demonstrated three stream computing technologies, as we all know, the most common ones: Storm, SparkStreaming and Flink. The tag for Storm is' past', the tag for SparkStreaming is' present', and the tag on Flink is' future'. At that time, our business was not processed in real time, so we didn't know anything about it. But I only remember the word 'future'.

Later, the processing related to real-time computing was added to the business, so I did some research on the three technologies of real-time computing before we started. Storm,SparkStreaming,FLink . In fact, I didn't do any relevant research, but based on the meeting at that time, I directly excluded Storm. Only between SparkStreaming and Flink is selected. And finally chose FLink.

Why Flink?

High throughput, low latency, high performance

For these three features, Flink is the only real-time processing framework in the community that supports all three features at the same time. While other SparkStreaming,Storm can not support three features at the same time, SparkStreaming is a feature of micro batch processing, so it can not achieve low latency protection, only high performance, high throughput. Storm can only support high performance and low latency.

Therefore, in the actual business use, a framework that ensures three features at the same time is very important for type selection.

Support for event time,process time,igest time

FLink supports event time, that is, the time of the data itself, which is very important for calculation and processing, and can prevent data calculation errors caused by out-of-order arrival. Keep the original timing of the data, and avoid the error of calculation results caused by network, hardware and so on.

On the other hand, the processing time and system time adopted by other systems may cause data calculation errors due to network, hardware, and even system startup problems.

Stateful computing

State management is included in Flink, which can store the intermediate result state of data calculation in memory or file, and the status information can be obtained when the next batch of events arrives. This will greatly improve the performance of the system because there is no need to recalculate again.

Flexible window mechanism

In real-time processing scenarios, the data is continuous. The scene of real-time processing also includes the processing of a range of data, such as one minute, 100 items and so on. Then Flink provides window mechanism to achieve flexible data cutting method, and provides a simple implementation scheme for 100 pieces of data or one minute calculation.

The windows provided by Flink are data-driven and time-driven. The window can be divided into scrolling window, tumbling window, session window and so on. Windows are freely combined to realize different data scenarios.

High fault tolerance

Flink provides a fault-tolerant mechanism, and the cluster anomalies caused by hardware, network and other problems in the data processing process can be recovered through the fault-tolerant mechanism. Fault tolerance ensures the exactly-once of data.

Research questions

In fact, the choice of Flink at that time is very unreasonable, at that time the mainstream real-time processing framework or SparkStreaming,Flink at that time the proportion is still very low, related books, documents are completely insufficient. I have no idea how many pits and how big there are in front of me. In the end, the relevant needs are realized by stumbling.

The research itself still needs to pay attention to whether the technology is market share. After all, it is very difficult for small companies to have Aliyun that can specifically select a small team to follow up on new technologies, or even pull branches to develop. The research technology itself may be more important to understand how many holes are in front of the technology. For the company, there is no perfect technology for the business, only the most suitable technology. For startups, it is more important to be able to achieve fast iteration, fast learning, fast mastery and someone to level the hole.

These are all the contents of the article "Why flink has become the hottest computing engine". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.