Shulou(Shulou.com)06/01 Report--

This article shows you a brief introduction to the Window window mechanism in Flink1.10, which is concise and easy to understand, which will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

I. the concept of Window

The above is a sentence from the official document of Flink1.10, which roughly means: Window is the core of dealing with infinite data flow, and Windows divides the flow into "buckets" of finite size, on which we can apply computation.

Flink believes that batch processing is a very special case of streaming. In streaming, we define a sliding window or scrolling window for the data and generate results each time the window slides or scrolls. Batch processing is different, we define a global window, all records belong to the same window; the underlying Flink engine is a streaming engine that implements streaming and batch processing. And window is a bridge from Streaming to Batch.

II. Window classification

1.keyed streams and non-keyed streams

The Flink program splits the data before calculating the window. There are two ways to control the flow: the first is keyed streams; and the second is non-keyed streams.

As can be seen from the figure above, for keyed streams, keyBy and window are called. First, the data stream is grouped according to key. The data flow passing through keyBy will form multiple groups of data, and multiple instances of downstream operators can be calculated in parallel, while non-keyed streams calls windowAll, which will not group the data flow, and all the data will be sent to a single instance downstream for processing. The principles of the two control flows are basically similar, and the only difference is that they are sent to multiple or single operators downstream for calculation.

2.Window Assinger

After the data is processed by the control flow, both control flows need to specify a window Assinger, which is responsible for assigning each incoming element to one or more windows. With window Assinger, various forms of window will be created to cover all kinds of scenarios we need. For our developers, we do not need to pay attention to window itself, we only need to pay attention to the classification of Window Assinger, so many videos about Flink do not explain the concept of control flow. Only talked about the classification of Window.

3.Window Assinger classification

Flink provides predefined Window Assinger programs for the most common use cases, that is, scrolling windows, sliding windows, session windows, and global windows. You can also implement custom window assigner by extending the Window Assigner class. All built-in window allocators (except global windows) assign elements to windows based on time, which can be either processing time or event time.

Window can be time-driven (Time Window, for example, every 60 seconds) or data-driven (Count Window, for example, every 200 elements). At the same time, windows driven by different events can be divided into the following categories: rolling window (Tumbling Window, no overlap), sliding window (Sliding Window, overlapping), session window (Session Window, active gap), global window. Below I will combine the ApacheFlink official website documents to explain to you these four windows.

3.1 Tumbling Window (scrolling window)

Scrolling window assigns each element to a window of the specified window size. The scroll window has a fixed size and the elements do not overlap. For example, if you specify a scrolling window with a size of 5 minutes, the current window is calculated and a new window starts every 5 minutes, as shown in the following figure.

3.2 Sliding Windows (sliding window)

The sliding window allocator assigns elements to fixed-length windows. Similar to the scroll window allocator, the window size is configured by the window size parameter. An additional window sliding parameter controls how often the sliding window starts. Therefore, if the sliding window is smaller than the window size, the sliding window can overlap. In this case, the element is assigned to multiple windows. For example, you can have a 10-minute window that can slide for 5 minutes. In this way, you will get a window containing events that have arrived within the last 10 minutes every 5 minutes, as shown in the following figure.

3.3 Session Windows (session window)

The session window allocator groups elements according to the active session. Compared with scrolling and sliding windows, session windows do not overlap and have no fixed start and end times. Conversely, when a session window does not receive an element for a period of time, it closes. When an inactive gap appears. The session window allocator can be configured as a static session interval or as a session interval extractor function, which defines the length of the period of inactivity. When this period expires, the current session is closed and subsequent elements are assigned to the new session window, as shown in the following figure:

3.4 Global Windows (Global window)

The global window allocator assigns all elements with the same key to the same global window. This window mode is useful only if you also specify a custom trigger. Otherwise, no calculation is performed because there is no global window that can handle the natural ends of aggregate elements, and all elements with the same keyed are assigned to a window, which is rarely used, as shown in the following figure:

At this point, some concepts of Flink's Window mechanism are introduced.

The above is a brief introduction to the Window window mechanism in Flink1.10. Have you learned any knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.