Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to understand the time mechanism in Flink1.10. The content of the article is of high quality, so the editor shares it for you as a reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

I. Overview

Here we introduce another core concept of Flink, the Event Time mechanism.

2. Three kinds of time mechanisms in Flink

Flink supports three concepts of time in stream processors, namely, EventTime, ProcessingTime, and IngestionTime,Flink. In streaming processing, most businesses use EventTime. Generally, only when EventTime cannot be used, consider other time attributes described below.

1.EventTime event time

EventTime is the time when the event occurred, which can be included in the event before the Flink stream handler is implemented, and the event timestamp can be extracted from each record.

In EventTime, the progress of time depends on the data, not any clock on the wall. The EventTime program must specify how the event EventTime Watermarks is generated, which is a signal mechanism for EventTime progress. This Watermarks mechanism is described in the following sections.

Assuming that all data has arrived, the event time operation will run as expected, producing correct and consistent results even when dealing with disordered or delayed events or reprocessing historical data. For example, the hourly event time window will contain all records that carry event timestamps belonging to that hour, regardless of the order in which they arrive or the time they are processed.

2.ProcessingTime processing time

ProcessingTime refers to the system time of the machine performing the corresponding operation. ProcessingTime is the default time concept of Flink. If you need to use other time types, you need to set it separately.

When the stream runs on processing time, all time-based operations, such as time windows, use the system clock of the machine running each operator. The hourly processing time window will include all records that reach a specific operator between the full hours indicated by the system clock. For example, if an application starts running at 9:15, the first hourly processing time window will include events processed between 9:15 and 10:00, the next window will include events processed between 10:00 and 11:00, and so on.

3.IngestionTime intake time

IngestionTime is the time when the event enters the Flink. In source operator, each record fetches the current time of the source in the form of a timestamp that is referenced by a time-based operation, such as a time window.

IngestionTime is conceptually located between EventTime and ProcessingTime. It is slightly more expensive than ProcessingTime, but provides more predictable results. Because IngestionTime uses a stable timestamp (allocated once at the source), different window operations on the record will reference the same timestamp, while in ProcessingTime, each window operator can assign the record to a different window (based on the local system clock and any transmission delay).

Compared to ProcessingTime, the IngestionTime program cannot handle any unordered events or delayed data, but the program does not have to specify how to generate Watermarks, because internally, it automatically assigns timestamps and automatically generates Watermarks.

The following is a picture of the time mark on Flink's official website:

4. Set time property

Typically, we set the stream processing time property when Flink initializes the streaming runtime. This setting is important because it determines how the data flow behaves. (for example, whether the event needs to be assigned a timestamp), and what type of time should be used for window operations; code example:

Final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment (); / / Flink defaults to ProcessingTimeenv.setStreamTimeCharacteristic (TimeCharacteristic.ProcessingTime); / / alternatively:// env.setStreamTimeCharacteristic (TimeCharacteristic.IngestionTime); / / env.setStreamTimeCharacteristic (TimeCharacteristic.EventTime)

At this point, the three time mechanisms of Flink have been explained, and we said that EventTime must specify how to generate the event time Watermarks.

On how to understand the time mechanism in Flink1.10 to share here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.