Shulou(Shulou.com)06/01 Report--

This article introduces the knowledge of "how to add connectors module to Flink". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

API in Flink

Flink provides different levels of abstraction for the development of streaming / batch processing applications.

The lowest abstraction of Flink API is stateful real-time stream processing. Its abstract implementation is Process Function, and Process Function is integrated into DataStream API by the Flink framework for our use. It allows users to freely handle events (data) from single or multiple streams in the application, and provides a state with global consistency and fault tolerance. In addition, users can register event time (event time) and processing time (processing time) callback methods in this layer of abstraction, allowing programs to implement complex calculations.

The second layer of abstraction of Flink API is Core APIs. In fact, many applications do not need to use the lowest abstraction of the above API, but can be programmed with Core APIs: it contains two parts: DataStream API (applied to bounded / unbounded data flow scenarios) and DataSet API (applied to bounded dataset scenarios). Streaming API (Fluent API) provided by Core APIs provides common module components for data processing, such as various forms of user-defined transformations (transformations), joins (joins), aggregations (aggregations), windows (window) and state (state) operations. The data types processed in this layer of API have their corresponding classes in each programming language.

The integration of low-level abstractions such as Process Function and DataStream API allows users to choose to use a lower-level abstract API to implement their requirements. DataSet API also provides additional primitives, such as loop / iteration (loop/iteration) operations.

The third layer of abstraction of Flink API is Table API. Table API is a table (Table)-centric declarative programming (DSL) API, such as in a streaming data scenario, which can represent a table that is changing dynamically. Table API follows the (extended) relational model: that is, tables have schema (similar to schema in relational databases), and Table API also provides operations similar to those in relational models, such as select, project, join, group-by, and aggregate. Table API programs define the logical operations that should be performed declaratively, rather than specifying exactly what code the program should perform. Although Table API is simple to use and can be extended by various types of user-defined functions, it is still less expressive than Core API. In addition, the Table API program optimizes user-written expressions using optimization rules in the optimizer before execution.

Tables and DataStream/DataSet can be switched seamlessly, and Flink allows users to mix Table API with DataStream/DataSet API when writing applications.

The top-level abstraction of Flink API is SQL. This layer of abstraction is similar to Table API in semantics and program expressions, but its program implementation is SQL query expressions. The relationship between SQL abstraction and Table API abstraction is very close, and SQL query statements can be executed on tables defined in Table API.

DataStream/DateSet API

The DataStream and DataSet programs in Flink are regular programs that transform data streams (for example, filtering, updating status, defining windows, aggregating). Data streams are initially created from various sources (for example, message queues, socket streams, files). The result is returned through the receiver, which can, for example, write data to a file or standard output (such as a command line terminal). Flink programs can be run in a variety of contexts, independently or embedded in other programs. Execution can be done in a local JVM or in a cluster of many computers.

Predefined Source and Sink

Some of the more basic Source and Sink are already built into Flink. The predefined data sources supports reading data from files, directories, socket, and collections and iterators. The predefined data sinks supports writing data to files, standard output (stdout), standard error output (stderr), and socket.

Official document

Https://ci.apache.org/projects/flink/flink-docs-release-1.12/zh/dev/connectors/

DataStream/DateSet API development

Starting from this article, add DataStream/DateSet API demonstration content, based on the original project, expand a connectors module; this module will demonstrate the simple use of the following components

Elasticsearch

File (text, csv)

Kafka

Jdbc (mysql)

Rabbitmq

Redis

New connectors module

In the current project, create a maven project module named connectors

Pom.xml

Connectors org.apache.flink flink-jdbc_2.11 1.10.1 mysql mysql-connector-java 5.1.47 org.apache.flink flink-connector-kafka_2.11 ${flink.version} org.apache.bahir flink-connector-redis_2.11 1.0 org.apache.flink flink-connector-rabbitmq_2.11 ${flink.version} org. Apache.flink flink-connector-elasticsearch7_2.11 ${flink.version}

Refresh the project maven and download the dependent component package for related functions

Create a user table (demonstrate use)

-- create a user table CREATE TABLE `tuser` (`id` int (8) NOT NULL AUTO_INCREMENT, `name` varchar (40) DEFAULT NULL, `age` int (3) DEFAULT NULL, `sex` int (2) DEFAULT NULL, `address` varchar (40) DEFAULT NULL, `createTime` timestamp NULL DEFAULT NULL, `createTimeSeries` bigint (20) DEFAULT NULL, PRIMARY KEY (`id`) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 under the database flink

Create an entity Bean (demonstrate use)

TUser.java

Package com.flink.examples;/** * @ Description t _ user table data wrapper class * / public class TUser {private Integer id; private String name; private Integer age; private Integer sex; private String address; private Long createTimeSeries; public TUser () {} public Integer getId () {return id;} public void setId (Integer id) {this.id = id } public String getName () {return name;} public void setName (String name) {this.name = name;} public Integer getAge () {return age;} public void setAge (Integer age) {this.age = age;} public Integer getSex () {return sex;} public void setSex (Integer sex) {this.sex = sex } public String getAddress () {return address;} public void setAddress (String address) {this.address = address;} public Long getCreateTimeSeries () {return createTimeSeries;} public void setCreateTimeSeries (Long createTimeSeries) {this.createTimeSeries = createTimeSeries } @ Override public String toString () {return "TUser {" + "id=" + id + ", name='" + name +'\'+ ", age=" + age + ", sex=" + sex + ", address='" + address +'\'+ " CreateTimeSeries= "+ createTimeSeries +'}' }}

TCount.java

Package com.flink.examples;/** * @ Description Statistics * / public class TCount {/ * gender * / private Integer sex; / * number * / private Integer num; public TCount () {} public TCount (Integer sex, Integer num) {this.sex = sex; this.num = num;} public Integer getSex () {return sex } public void setSex (Integer sex) {this.sex = sex;} public Integer getNum () {return num;} public void setNum (Integer num) {this.num = num;}}

Engineering module

This is the end of the introduction of "how to add connectors modules to Flink". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.