Shulou(Shulou.com)06/01 Report--

Flink is how to ensure end-to-end exactly-once semantics, many novices are not very clear about this, in order to help you solve this problem, the following small series will explain in detail for everyone, people who have this need can learn, I hope you can gain something.

In December 2017, Apache Flink 1.4.0 was released. There is one landmark feature: Two Phase CommitSink Functions.

TwoPhaseCommitSinkFunction divides the last logic written to storage into two commits, making it possible to build an end-to-end exactly-once semantic flink application from the data source to the data output. Of course, the data output of TwoPhaseCommitSinkFunction includes versions of apache kafka 0.11 and above. Flink provides an abstract TwoPhaseCommitSinkFunction class that allows developers to implement end-to-end exactly-once semantics with less code.

Next, we further introduce this feature of flink:

·The role of flink checkpoints in guaranteeing exactly-once semantics·Flink is how to guarantee exactly-once semantics from data source to data output through a two-part commit protocol·An example to explain how to implement an exactly-once sink using TwoPhaseCommitSinkFunction

Exactly-once semantics ensure that there is no data loss or duplication in the final data processing result and data intake. The checkpoint of flink contains the current state of the flink application and the location of the data input stream (offset for kafka). Checkpoints can be asynchronously persisted to storage systems such as s3 or hdfs. If the flink application fails or is upgraded, the state in the checkpoint can be pulled to restore the data from the last failure. Prior to flink 1.4.0, flink guaranteed exactly-once semantics within flink applications through checkpoints. TwoPhaseCommitSinkFunctio is now added to guarantee end-to-end exactly-once semantics. The two-commit approach to semantics requires that the external system to which flink is connected supports two-part commit, i.e. the external system supports the feature that data can be precommitted and rolled back without a final commit. Later we will see how flink is a two-commit protocol with external systems to ensure semantics using flink to ensure that end-to-end data is not lost and not duplicated. Let's take a look at the example of flink consuming and writing kafka to ensure exactly-once semantics through two commits. kafka supports transaction operations since 0.11. To use flink end-to-end exactly-once semantics, kafka of flink's sink needs to be version 0.11 or higher. DELL/EMC's Pravega also supports the use of flink to ensure end-to-end exactly-once semantics. This example includes the following steps:

·Read data from kafka·An aggregation window operation·Write data to kafka

To guarantee exactly-once, all operations written to kafka must be things. Batch commit data between checkpiont so that uncommitted data can be rolled back after a task fails. However, a simple commit and rollback is not enough for a distributed streaming data processing system.

The first step in the two-part commit protocol is pre-commit. Flink's jobmanager inserts a checkpoint tag into the data stream (this tag can be used to distinguish the data at this checkpoint from the data at the next checkpoint). This tag is passed through the dag. Each operator in dag encounters this tag and triggers a snapshot of the operator's state.

The operator that reads kafka stores the offset of kafka when a checkpoint marker is encountered. This checkpoint tag is then passed to the next operator. Next comes flink's memory operator. These inner operators do not need to be considered in the two-part commit protocol because their state updates or rolls back with the state of the flink as a whole.

When dealing with external systems, a two-step commit protocol is required to ensure that data is not lost or duplicated. Under this step, all data submitted to kafka is pre-submitted.

When the snapshot of all operators is complete, that is, when the checkpoint is completed, Flink's jobmanager will notify all operators that the checkpoint is complete, and Flink's operators responsible for writing data to kafka will officially submit the data written before. Failure at any stage of the task will revert from the previous state, and any data that has not been formally committed will be rolled back.

5 To summarize Flink's two-step submission:

When all operators complete their snapshots, a formal commit operation occurs·When any subtask fails in the pre-commit phase, the other tasks stop immediately and roll back to the state of the last successful snapshot. After the pre-commit state is successful, the external system needs to perfectly support the operations prior to formal commit. If a commit failure occurs, the entire

The flink app will enter a failed state and restart, after which it will continue to attempt commit operations from the previous state. Applying the two-step commit operator in flink When using the two-step commit operator, we can inherit the virtual class TwoPhaseCommitSinkFunction.

This virtual class is explained by a simple example of writing a file. This two-step commit class has four states.

Begin Transaction-Create a temporary folder to write data to. PreCommit-writes cached data in memory to a file and closes it. Commit-Place previously written temporary files in the destination directory. This means that there will be some delay in the final data. Abort-Discard temporary files.

If the failure occurs after successful pre-commit, before formal commit. Pre-committed data can be submitted or deleted based on status.

Summary Flink guarantees end-to-end exactly-once semantics through state and two-commit protocols. In batch processing, flink does not have to persist every computation into memory. Flink supports the producer's exactly-once semantic guarantee above Pravega and kafka 0.11.

Did reading the above help you? If you still want to have further understanding of related knowledge or read more related articles, please pay attention to the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.