Shulou(Shulou.com)05/31 Report--

Editor to share with you what the external data sources of Apache Flink Table API and SQL API are, I believe most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to know it!

"the Table API and SQL API implementations of Apache Flink connect external data sources directly to Table connector. Tabel connector reads external data sources into Flink for calculation or writes Apache Flink results to external data sources."

Table API and SQL API of Apache Flink have mentioned the way of Table Source and Table Sink to realize the registration of the data source in the result writing and so on. Apache Flink has basically realized the requirement of registering and writing data sources, but it is still not flexible enough. Users may prefer to choose different data sources through configuration and write data. So Apache Flink also provides Table connector.

The implementation of Table connector can not only better API and SQL Client, but also Table connector registers the data in the same way as CREATE TABLE, which can write the data to different locations without modifying the code. The Tabel connector method specifies the Descriptor,withFormat method that needs to connect to the Table connector and specifies the file format for output or input (csv,JSON,Parquet, etc.). The withSchema method specifies the table structure registered in TableEnvironment.

TableEnv.connect (...)

.withFormat (...)

.withSchema (...)

.inAppendMode () / / Update mode

.registerTableSource ("MyTable")

After registering the data source in the above way, we can read or write the data directly to the table. That is to say, we can manipulate the DBMS table to read and write data similar to that of the table.

In many current cloud scenarios, such as Aliyun and Huawei Cloud, real-time computing products are provided. Products use SQL to register, register other products and storage formats of specified cloud vendors, and specify the read data structure withSchema and updated mode to achieve data reading or writing of calculation results.

CREATE TABLE datahub_stream (

Name VARCHAR

Age BIGINT

Birthday BIGINT

) WITH (

Type=' products'

EndPoint='...'

Project='...'

Topic='...'

...

);

The steps involved in our use are also roughly divided into these, as shown in the following code. The connect method specifies the file format of the output or input specified by the Desciriptor,withFormat corresponding to the connection, such as JSON,CSV,Avro, and so on. WithSchema is used to specify the table structure registered in TableEnvironment. InAppendMode specifies the update mode of the data. Finally, the external data source is registered to TableEnvironment through the registerTableSource method for query calculation.

TalbEnv.connect (...)

.withFormat (...)

.withSchema (...)

.inAppendMode ()

RegisterTableSource ("TableName") above is all the content of the article "what are the external data sources of Apache Flink Table API and SQL API?" Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.