Example Analysis of Flink DataSet programming in Apache Flink

This article introduces to you what is Flink DataSet programming in Apache Flink, the content is very detailed, interested friends can refer to, hope to be helpful to you.

DataSet programming in Flink is very conventional programming, which only needs to implement the transformation of his dataset (such as filtering, mapping, joining, grouping). This dataset is originally created by a data source (for example, reading files, loading a local collection of local datasets), and the results of the transformation are returned to the local (or distributed) file system or terminal via sink. Flink programs can be run in a variety of environments, such as standalone, or embedded in other programs. The execution process can be in the local JVM or in the cluster.

Source = = > Flink (transformation) = > Sink

File based

ReadTextFile (path) / TextInputFormat-Reads files line wise and returns them as Strings.

ReadTextFileWithValue (path) / TextValueInputFormat-Reads files line wise and returns them as StringValues. StringValues are mutable strings.

ReadCsvFile (path) / CsvInputFormat-Parses files of comma (or another char) delimited fields. Returns a DataSet of tuples or POJOs. Supports the basic java types and their Value counterparts as field types.

ReadFileOfPrimitives (path, Class) / PrimitiveInputFormat-Parses files of new-line (or another char sequence) delimited primitive data types such as String or Integer.

ReadFileOfPrimitives (path, delimiter, Class) / PrimitiveInputFormat-Parses files of new-line (or another char sequence) delimited primitive data types such as String or Integer using the given delimiter.

Based on set

FromCollection (Collection)

FromCollection (Iterator, Class)

FromElements (T...)

FromParallelCollection (SplittableIterator, Class)

GenerateSequence (from, to)

Create a DataSet from a simple collection-based

Collection-based data sources are often used in the development environment or in programmers' learning, and we can create the data we need at will, because the way is simple. Let's use collections as data sources in both java and scala ways. The data source is simple from 1 to 10

Javaimport org.apache.flink.api.java.ExecutionEnvironment;import java.util.ArrayList;import java.util.List;public class JavaDataSetSourceApp {public static void main (String [] args) throws Exception {ExecutionEnvironment executionEnvironment = ExecutionEnvironment.getExecutionEnvironment (); fromCollection (executionEnvironment);} public static void fromCollection (ExecutionEnvironment env) throws Exception {List list = new ArrayList (); for (int I = 1; I