Shulou(Shulou.com)06/01 Report--

This article mainly explains "how to use Flink Reduce". The content in the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn how to use Flink Reduce.

Reduce operator: performs a rolling aggregation calculation on the data stream and returns the merged result of each rolling aggregation calculation

Sample environment

Java.version: 1.8.xflink.version: 1.11.1

Sample data source (project code cloud download)

Building Development Environment and data of Flink system example

Reduce.java

Import com.flink.examples.DataSource;import org.apache.flink.api.common.functions.ReduceFunction;import org.apache.flink.api.java.functions.KeySelector;import org.apache.flink.api.java.tuple.Tuple3;import org.apache.flink.streaming.api.datastream.KeyedStream;import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;import java.util.List / * * @ Description Reduce operator: performs a rolling aggregate calculation on the data stream and returns the merged result of each rolling aggregate calculation * / public class Reduce {/ * traversing the collection. The partition prints the result of each rolling aggregation * @ param args * @ throws Exception * / public static void main (String [] args) throws Exception {final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment () Env.setParallelism (4); List tuple3List = DataSource.getTuple3ToList () / / Note: when using Integer for partition, the partition result will be incorrect. Convert to String type output key to correctly output KeyedStream keyedStream = env.fromCollection (tuple3List) .keyby (new KeySelector () {@ Override public String getKey (Tuple3 tuple3) throws Exception {/ / F1) as the gender field, and partition return String.valueOf (tuple3.f1) with the same F1 value (gender). }}); SingleOutputStreamOperator result = keyedStream.reduce (new ReduceFunction () {@ Override public Tuple3 reduce (Tuple3 t0, Tuple3 T1) throws Exception {int totalAge = t0.f2 + t1.f2; return new Tuple3 (", t0.f1, totalAge);}}); result.print () Env.execute ("flink Reduce job");}}

Print the result

# # explanation: why every parameter of the first data object in each partition has a value, because the scrolling aggregate returns the first data object superimposed forward from the second data object to start the calculation, so the first data object does not enter the reduce method at all 2 > (Zhang San, man,20) 2 > (, man,49) 2 > (, man,79) 4 > (Li Si, girl,24) 4 > (, girl,56) 4 > (, girl,74) Thank you for your reading, the above is the content of "how to use Flink Reduce". After the study of this article, I believe you have a deeper understanding of how to use Flink Reduce, and the specific use needs to be verified by practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.