Shulou(Shulou.com)06/03 Report--

DataSet Api programming guide is what, many novices are not very clear, in order to help you solve this problem, the following editor will explain for you in detail, people with this need can come to learn, I hope you can gain something.

ApacheFlink is a distributed processing framework with high throughput, low latency and high performance. Today, with the rise of real-time computing, Flink is developing rapidly. Due to the advantages of performance and the characteristics of batch processing and streaming processing, Flink may be subverting big data's ecology.

DataSetAPI

To run Flink first, we need to download and extract the Flink binaries.

We can choose the combined version of Flink and Scala, and here we choose the latest version 1.9 ApacheFlink1.9.0forScala2.12 to download.

After the download is successful, you can run Flink through Windows's bat file or Cygwin on the windows system.

In the linux system, it is divided into stand-alone, cluster, Hadoop and other situations.

Flink programming model, Flink provides different levels of abstraction to develop streaming or batch applications. In this article, we introduce the most commonly used batch programming model of DataSetAPI,Flink.

The DataSet program in Flink is a regular program that implements dataset transformation (for example, Filter, mapping, join, grouping). Datasets are originally created from some sources (for example, by reading files or from local collections). The result is returned through the receiver, which can, for example, write (distributed) data to a file or standard output (such as a command line terminal). Flink programs can be run in a variety of environments, independently or embedded in other programs. Execution can be performed on a local JVM or on a cluster of many computers.

Data set conversion

A data transformation converts one or more DataSet into a new DataSet. Programs can combine multiple transformations into complex assemblies.

The most important operators in DataSetAPI are these operators. After we access the data, we process the data through these operators and get the results we want.

Collect data sources and receivers

It is troublesome to complete the input and check the output of the analyzer by creating the input file and reading the output file. Flink has special data sources and receivers, supported by the Java collection to simplify testing. Once the program has been tested, the source and receiver can be easily replaced by the source and receiver that read / write to an external data store, such as HDFS.

In development, we often use the receiver directly to receive the data source.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.