Shulou(Shulou.com)06/03 Report--

Bypass output (side output)

In addition to the mainstream result output from the data flow operator,    can produce any number of stream bypass output results. The bypass output result data type can be completely different from the mainstream result data type and other bypass output result data types. This operator is very useful when you need to split the data stream. You usually need to copy the stream and then filter out the unwanted data from each data stream.

   when using bypass output, you first need to define an OutputTag to identify a bypass output stream.

Java

/ / this needs to be an anonymous inner class, so that we can analyze the typeOutputTag outputTag = new OutputTag ("side-output") {}

Scala

Val outputTag = OutputTag [String] ("side-output")

   notices how OutputTag is based on the element type typed contained in the bypass output stream.

   can transmit data to bypass output through the following function.

ProcessFunctionCoProcessFunctionProcessWindowFunctionProcessAllWindowFunction

   can use the Context parameter (exposed to the user in the above function) to send data to the bypass output identified by OutputTag. The following is an example of bypass output data from ProcessFunction:

Java:

DataStream input =...; final OutputTag outputTag = new OutputTag ("side-output") {}; SingleOutputStreamOperator mainDataStream = input .process (new ProcessFunction () {@ Override public void processElement (Integer value, Context ctx, Collector out) throws Exception {/ / emit data to regular output out.collect (value); / / emit data to side output ctx.output (outputTag, "sideout-" + String.valueOf (value)) })

Scala:

Val input: DataStream [Int] =... val outputTag = OutputTag [String] ("side-output") val mainDataStream = input .process (new ProcessFunction [Int, Int] {override def processElement (value: Int, ctx: ProcessFunction [Int, Int] # Context, out: Collector [Int]): Unit = {/ / emit data to regular output out.collect (value) / / emit data to side output ctx.output (outputTag) "sideout-" + String.valueOf (value))}})

To read the bypass output stream,    uses getSideOutput (OutputTag) after the data stream operation. You will get the result of typing the bypass output stream.

Java:

Final OutputTag outputTag = new OutputTag ("side-output") {}; SingleOutputStreamOperator mainDataStream =...; DataStream sideOutputStream = mainDataStream.getSideOutput (outputTag)

Scala:

Val outputTag = OutputTag [String] ("side-output") val mainDataStream =... val sideOutputStream: DataStream [String] = mainDataStream.getSideOutput (outputTag)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.