Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to correctly use FlinkX, the content of the article is of high quality, so the editor will share it with you for reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

Git Clone project

First download the project from Github and import it into IDEA (there are many methods, introduce a common one)

From the IDEA menu bar, Git option-> Clone-> Flinkx project address, click Clone to get a copy of the Flinkx source code!

Version correspondence

The default version of the project clone is 1.10_release, and the recommended stable versions are 1.8_release and 1.10_release. The latest version is 1.11_release. But! The latest doesn't mean the most stable. Each version of Flinkx corresponds to the version of Flink, and the upper and lower versions are not necessarily compatible, so please make sure the corresponding version of Flinkx and Flinkx before using it.

Project compilation

Because there are so many plug-ins involved in the project, it takes a lot of time to compile the entire project, so it is recommended that you comment out plug-ins that do not need to be used. At the same time, it is strongly recommended to look at the official documentation before compiling

!!! However, please pay attention! Attention, please! Attention, please! ⚠️

1. Some plug-ins have dependencies, such as:

Flinkx-core is dependent on all plug-ins and must not be commented!

The flinkx-rdb module is dependent on all rdb data source plug-ins, so in scenarios like mysql,oracle (including hive, because jdbc is used to connect), the flinkx-rdb module must not be annotated!

Flinkx-test is a module used to debug code locally, and the project must be commented out when packaged into a jar package!

The flinkx-launcher module is used to submit executable tasks to the corresponding execution environment and cannot be commented!

two。 Some jar packages in some plug-ins may not be available in the maven central repository, but in the project's jars directory, developers have kindly prepared jar.

In addition, an executable script is specially written to install the jar package to the local repository. (how to execute scripts under different systems, please do it yourself.)

[the picture is taken from the official document]

Please pay attention to the above! Attention, please! Attention, please! ⚠️

Project compilation command

Mvn clean package-Dmaven.test.skip

Then, under the root directory of the project, generate the corresponding plug-in directory

1.8_release-> plugins

1.10_release-> syncplugins

II. Task submission

Flinkx supports task submission in local-test (submit tasks through flinkx-test module), local, standalone, yarn-session and yarn-per-job mode, while application mode submission is not supported.

Submit a task from idea

The version of idea used is 2020.3 public version, which can be modified by itself in some different places.

Here, take the yarn-per-job mode as an example. Other modes are similar. You can configure the task submission parameters by yourself in the official documentation.

1. Configure idea-application

Module: flinkx-launcher

Entry class: Launcher

There is also a shortcut. Directly in the entry class, under the execution button of the class, select "Modify Run Configuration" to modify it.

You can mainly modify the task parameters according to your own situation. The task parameters that have been used in idea are posted below.

-mode yarnPer-name flinkx-test-job docs/example/stream_stream.json-pluginRoot syncplugins-flinkconf / dtstack/conf/flink-yarnconf / dtstack/conf/yarn-flinkLibJar / dtstack/flink-1.10.1/lib-confProp {\ "flink.checkpoint.interval"\: 60000}-queue c

What is the specific meaning of the task parameters? Please see the parameter description of the official document.

How to configure the task JSON? Please take a look at the official documentation, there is no more to say here, moreover, the official also specially collates the task of commonly used plug-ins JSON, as long as the corresponding modify parameters, click on the link to jump (who else is such a sweet official? Isn't it worth a star? )

Third, debug code

Local debugging

First open the flinkx-test module

In LocalTest, jobPath is changed to the absolute path of the task JSON. Click execute to debug locally. Officials have also specially added commonly used parameter configurations.

Remote debugging

If you need remote debugging, you need to add the remote debugging configuration of Flink in flink-conf.yaml, then configure "JVM Remote" in idea, and break points in the code block (this method can also debug the code of Flink itself)

Env.java.opts.jobmanager:-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005 env.java.opts.taskmanager:-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5006

You only need to modify these two places marked. If it is a HA cluster, you need to modify it according to the log (how to look at the log, how to modify it, and check the information yourself).

On how to correctly use FlinkX to share here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.