Shulou(Shulou.com)06/01 Report--

The main content of this article is to explain "what is the reason for choosing Apache Flink in large data stream processing". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn "what is the reason for choosing Apache Flink in large data stream processing?"

With the rapid development of big data's technology in recent years, people have higher and higher requirements for data processing, from the earliest MapReduce to the later hive, and then to the later spark. In order to obtain faster and more timely results, the computing model is also slowly changing from the previous offline data to streaming processing, such as the real-time big screen of Singles Day Ali every year, which requires second output results. For example, when we are driving at 100 mph, we want the map navigation software to give us millisecond delay navigation information.

So why do we choose Apache Flink as our streaming framework when we already have streaming frameworks like storm and spark streaming?

Real stream processing with low latency

For spark streaming, although it is also a stream processing framework, its underlying layer is a micro-batch mode, but this batch is small enough to make us look like a stream processing, which is sufficient for our ordinary needs, but for the map navigation software we mentioned above, the delay we need is millisecond, because if you delay half a minute I may have driven a long way, and the navigation information you gave me is useless.

Therefore, for the framework of micro-batch processing, it will naturally cause data delay. Flink, as a real stream processing framework, can process one for each data processing to achieve real stream processing and low latency.

High huff and puff

As we said earlier, Ali Singles Day data calculation is very large, at this time to calculate such a large amount of data, we need to have a computing framework that supports high throughput to meet more real-time needs.

Multiple windows

Flink itself provides a variety of flexible windows, let's talk about the meaning of these windows.

Scroll window: calculate the total sales for the current five minutes every five minutes. Slide window: calculate the total sales for the previous hour every five minutes. Session window: statistics of the total number of visits made by the user during the time he logged in. Global window: we can count some values since the program was launched.

In addition to the time window (time window), there is also a count window (count window), the count window window can also have scroll and slide windows, for example, we count the average of these 100 numbers every 100 numbers.

Bring your own status (state)

What is status, to put it in vernacular, for example, we consume a piece of data from kafka, and then write it to a file one by one. This is stateless calculation, because a single piece of data does not need to rely on the data before and after it.

When we want to implement a window count and count the number of pv per hour, we can imagine that there is a variable that adds one to each data, and then when the program is running halfway, it dies for some reason. At this time, if that variable is stored in memory, it will be lost. After the program restarts, we have to re-calculate it from the beginning of the window. Is there a mechanism? Can automatically help me to save this temporary variable reliably, this is the state in flink, for the above scenario, when we restore the program, we choose to recover from the previous checkpoint, so we can continue to calculate from the time the program hangs, instead of calculating from the beginning of the window.

Precise primary transmission semantics

For a large distributed system, it is very common for programs to fail due to network, disk and other reasons, so when we restore the program, how to ensure that the data is not lost or heavy?

Flink provides Exactly-once semantics to deal with this problem.

time management

Flink provides a variety of time semantics for our use.

Event time

That is, we use the time in the data when we calculate, for example, our program has hung up for half an hour for some reason, and when the program gets up, we hope that the program can continue to process the last time, then the event time will come in handy.

In addition, for some alarm systems, the time in the log can really reflect the problematic time, which is more meaningful.

Processing time

That is, the current time of the flink program

Uptake time

The time when the data entered the flink program

Watermark

In a real production environment, data transmission will go through many processes. In this process, it is inevitable that the data will arrive late due to network jitter and other reasons, and the data that should have come first is late. How to deal with this situation? flink's watermark mechanism will help you deal with it.

We can simply understand that by setting an acceptable delay time, if your data does not come to the point, flink will wait for you for a few seconds, and then wait for your data to come and then trigger the calculation, but because it is stream processing, certainly can not wait indefinitely, for the data that has not come yet beyond the waiting time I set, then I can only abandon or save it to another stream to deal with with other logic.

Complex event processing

Let's start with such a scenario, for example, if we want to monitor the temperature of the machine, we will generate a warning if the temperature exceeds 50 degrees three times in 10 minutes, and if there are two such warnings in an hour in a row, an alarm will be generated.

For such a scenario, do you feel that ordinary api programs are not easy to do? Well, flink's complex event processing (CEP) comes in handy, and cep can be used to deal with many similar complex scenarios.

At this point, I believe you have a deeper understanding of "what is the reason for choosing Apache Flink in large data stream processing". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.