Shulou(Shulou.com)06/01 Report--

This article mainly explains "how to use split in Flink". The content in the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn how to use split in Flink.

The Magic Divider of flink-sideoutput

This can be used for diversion, and the data can be filtered and returned at a convenient time.

There are also iterations for algorithm processing, which we have talked about in two articles:

Flink-specific iterative operation-bulkIteration

The DeltaI iterative operation of Flink Dataset has to be met.

One is full iteration and the other is incremental iteration.

There is also an excellent and chicken-ribbed watermark mechanism.

Don't understand watermark? Come on.

In fact, there is another word about iterative operation, that is, iterative operation of stream processing. Then this article will analyze this.

Flink's iterative flow program actually implements a step function and then embeds it inside the IterativeStream. Be aware that Flink's Datastream does not normally end, so there is no maximum number of iterations. In this case, you need to specify which type of data needs to be reflowed to continue iteration, and which type of data continues to be transferred downwards. There are two ways to divert this type of data: split and filter. The official website uses filter when introducing iterative streams. Here we will first follow the introduction of the official website, and then use split to make a demo for you when the case is demonstrated.

First, create an IterativeStream

IterativeStream iteration = input.iterate ()

Then you can define the logical operation to be done, and here is a simple example of map on the official website.

DataStream iterationBody = iteration.map (/ * this is executed many times * /)

Call the closeWith (feedbackStream) method of IterativeStream to perform closed-loop operations on the iterative flow. The DataStream passed to the closeWith function returns the header of the value iteration. A common practice is to use filter to separate the backward iterative part of the flow from the forward part of the flow.

Iteration.closeWith (iterationBody.filter (/ * one part of the stream * /))

DataStream output = iterationBody.filter (/ * some other part of the stream * /)

Officials give an example of a continuous minus of 1 until the data is zero:

DataStream someIntegers = env.generateSequence (0, 1000)

/ / create an iterative flow

IterativeStream iteration = someIntegers.iterate ()

/ / add processing logic and subtract one from the element.

DataStream minusOne = iteration.map (new MapFunction () {

@ Override

Public Long map (Long value) throws Exception {

Return value-1

}

});

/ / get the stream to be iterated

DataStream stillGreaterThanZero= minusOne.filter (new FilterFunction () {

@ Override

Public boolean filter (Long value) throws Exception {

Return (value > 0)

}

});

/ / A pair of flows that need to be iterated form a closed loop

Iteration.closeWith (stillGreaterThanZero)

/ / data less than or equal to 0 continues to be transmitted forward

DataStream lessThanZero = minusOne.filter (new FilterFunction () {

@ Override

Public boolean filter (Long value) throws Exception {

Return (value

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.