Shulou(Shulou.com)06/01 Report--

This article is about what AllowedLateness is in Flink. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

Flink AllowedLateness

AllowedLateness is used in Flink window to tolerate late data for a certain period of time. The easiest misunderstanding about allowedLateness is that it is equivalent to subtracting or postponing watermark for a certain period of time, and tolerates late data for a certain period of time by delaying window triggering. In fact, allowedLateness does not delay the window trigger time, it changes the window cleaning time (cleanupTime). Here are org.apache.flink.streaming.runtime.operators.windowing.WindowOperator 's cleanupTime and registerCleanupTimer, which indicate the cleanup of the allowedLateness influence window:

/ * * Returns the cleanup time for a window, which is * {@ code window.maxTimestamp + allowedLateness}. In * case this leads to a value greater than {@ link Long#MAX_VALUE} * then a cleanup time of {@ link Long#MAX_VALUE} is * returned. * * @ param window the window whose cleanup time we are computing. * / private long cleanupTime (W window) {if (windowAssigner.isEventTime ()) {long cleanupTime = window.maxTimestamp () + allowedLateness; return cleanupTime > = window.maxTimestamp ()? CleanupTime: Long.MAX_VALUE;} else {return window.maxTimestamp ();}} / * * Registers a timer to cleanup the content of the window. * @ param window * the window whose state to discard * / protected void registerCleanupTimer (W window) {long cleanupTime = cleanupTime (window); if (cleanupTime = = Long.MAX_VALUE) {/ / don't set a GC timer for "end of time" return;} if (windowAssigner.isEventTime ()) {triggerContext.registerEventTimeTimer (cleanupTime) } else {triggerContext.registerProcessingTimeTimer (cleanupTime);}}

In other words, allowedLateness changes the window cleanup time. After watermark triggers window calculation, the window is not cleared immediately but waits for allowedLateness to be cleared. During this period, a window calculation is triggered when late data is received from the window. As shown in the figure below, a small circle represents the elements in the window. When watermark reaches the window endTime, it will trigger a window calculation, and a late circle will trigger a window calculation respectively.

Thank you for reading! This is the end of this article on "what is AllowedLateness in Flink?". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.