Shulou(Shulou.com)06/02 Report--

This article will explain in detail how to use the ProcessFunction class in Flink, the content of the article is of high quality, so the editor will share it for you as a reference. I hope you will have some understanding of the relevant knowledge after reading this article.

Flink processing function actual combat series link

In-depth understanding of ProcessFunction's status operation (Flink-1.10)

ProcessFunction

KeyedProcessFunction class

ProcessAllWindowFunction (window handling)

CoProcessFunction (dual stream processing)

About handlers (Process Function)

As shown in the figure below, in conventional business development, SQL, Table API and DataStream API are more commonly used, while Porcession in Low-level is relatively less used. from this chapter, we will be familiar with the processing function (Process Function) through actual combat to see what capabilities this series of low-level operators can bring us.

About the ProcessFunction class

There are many kinds of processing functions, the most basic of which should be the ProcessFunction class. Take a look at its class diagram. We can see that there are the characteristics of RichFunction, open and close, and then we have two important methods, processElement and onTimer: the common features are as follows:

Working with individual elements

Access timestamp

Bypass output

Next, write two applications to experience the above functions.

Version information

Development environment operating system: MacBook Pro 13 inch, macOS Catalina 10.15.3

Development tools: IDEA ULTIMATE 2018.3

JDK:1.8.0_211

Maven:3.6.0

Flink:1.9.2

Source code download

If you don't want to write code, the source code for the entire series can be downloaded from GitHub. The address and link information are shown in the following table (https://github.com/zq2599/blog_demos):

Name Link Note Project Home Page https://github.com/zq2599/blog_demos the home page of the project on GitHub git warehouse address (https) https://github.com/zq2599/blog_demos.git the warehouse address of the project source code, https protocol git warehouse address (ssh) git@github.com:zq2599/blog_demos.git the warehouse address of the project source code, ssh protocol

There are multiple folders in this git project. The application of this chapter is under the flinkstudy folder, as shown in the red box below:

Create a project

Execute the following command to create an application project for flink-1.9.2:

Mvn\ archetype:generate\-DarchetypeGroupId=org.apache.flink\-DarchetypeArtifactId=flink-quickstart-java\-DarchetypeVersion=1.9.2

Enter groupId:com.bolingcavalry,architectid:flinkdemo as prompted

The first demo

The first demo is used to experience the following two features:

Working with individual elements

Access timestamp

Create a Simple.java as follows:

Package com.bolingcavalry.processfunction;import org.apache.flink.api.java.tuple.Tuple2;import org.apache.flink.streaming.api.TimeCharacteristic;import org.apache.flink.streaming.api.datastream.DataStream;import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;import org.apache.flink.streaming.api.functions.ProcessFunction;import org.apache.flink.streaming.api.functions.source.SourceFunction;import org.apache.flink.util.Collector Public class Simple {public static void main (String [] args) throws Exception {final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment (); env.setStreamTimeCharacteristic (TimeCharacteristic.EventTime); / / parallelism is 1 env.setParallelism (1) / / set the data source, with a total of three elements DataStream dataStream = env.addSource (new SourceFunction () {@ Override public void run (SourceContext ctx) throws Exception {for (int item1; I)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.