Shulou(Shulou.com)06/01 Report--

This article mainly explains "what are the trigger knowledge points of flink". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Next, let the editor take you to learn "what are the trigger knowledge points of flink"?

A small function of flink window calculation-trigger.

The concept of window is relatively easy to understand, that is, select a range on a certain latitude (time, number of bars, etc.), this range is the window, and then our later calculation is for this window. Today's example is mainly about the time window.

I have to say that the window operations supported by flink are relatively rich:

Session window, scrolling window, sliding window, etc.

Take the sliding window as an example. The name of the sliding window means that in addition to opening the window in the time dimension, we also need to specify what steps the window should take to move on the time scale. This is called the sliding interval, as shown below:

Thus, it can be obtained that the two important parameters of sliding window are window size and sliding interval.

For the window operation of flink, especially the window operation based on event time, we also need to handle three important knowledge points:

Window allocator: it determines which window the data that flows into flink belongs to.

Timestamp extractor / watermark generator: extracts timestamps and drives the normal execution of the program.

Trigger: determines when the data will land.

We will explain all of these later on the planet. This article is mainly about trigger.

Recall that the window operation of spark streaming is actually a larger batch, in which the window slides at a specified sliding interval, and then the size of the window and sliding interval is an integral multiple of the batch.

When will the data be processed and landed?

The sliding interval is the period of calculation. Then the processing time is the time from the start of the window to the data landing, which is normally less than or equal to the sliding interval, otherwise queue squeezing will occur.

In fact, this idea is also spark streaming's idea of micro-batch processing, and it is also based on the idea of processing time.

But flink is not, flink is real real-time processing, after data flows into flink's source, if we need a window function, we have to use certain rules to judge or call to determine which window the data should belong to, and then if the window is based on the event time, we should also provide a timestamp extractor and watermark allocator, and finally specify what conditions are met to trigger window calculation and output results.

That may say that the trigger window calculation, is not the time to the end of the window directly output on the line? In this way, the frequency of the output can be directly controlled by the sliding interval.

In fact, no, based on the event time processing mechanism, the data will lag in some unexpected situations, such as forward failure, etc., in this case, for flink, we can set some parameters to allow processing lagging elements, such as allowing it to lag by an hour, then the window output interval actually needs to be added at this time. At this time, if we want as much real-time output as possible, we need to use flink's trigger mechanism.

Trigger defines when to use the window calculation function to calculate the window. Each window allocator has a default Trigger. If the default Trigger does not meet your needs, you can specify a custom trigger ().

The trigger interface has five methods that allow trigger to react to different events:

OnElement (): this method is called by every element that enters the window.

OnEventTime (): the event time is called when the timer is triggered.

OnProcessingTime (): processing time is called when timer is triggered.

OnMerge (): stateful triggers are related and merge the state of the two triggers when their corresponding windows are merged, such as using the session window.

Clear (): this method mainly performs the delete operation of the window.

There are two points to pay attention to about the above methods:

1)。 The first three methods determine how to manipulate input events by returning a TriggerResult.

CONTINUE: nothing.

FIRE: triggers the calculation.

PURE: clears the elements of the window.

FIRE_AND_PURE: triggers calculation and clears window elements.

2)。 Any of these methods can be used to register processing or event time timers for future operations.

Built-in and custom triggers

There are some built-in triggers inside Flink:

EventTimeTrigger: trigger the window based on the event time and watermark mechanism.

ProcessingTimeTrigger: triggered based on processing time.

CountTrigger: calculation is triggered if the number of window elements exceeds a predetermined limit.

PurgingTrigger, as a parameter of other trigger, converts it into a purging trigger.

The default trigger for WindowAssigner applies to many cases. For example, all event time-based window allocators use EventTimeTrigger as the default trigger. The trigger directly triggers the calculated output when the watermark reaches the window's deadline.

At this point, I believe you have a deeper understanding of "what are the trigger knowledge points of flink?" you might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.