Network Security Internet Technology Development Database Servers Mobile Phone Android Software Apple Software Computer Software News IT Information

In addition to Weibo, there is also WeChat

Please pay attention

WeChat public account

Shulou

Big data's Apache Beam uses Flink Runner pipe parameters

2025-04-05 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Internet Technology >

Share

Shulou(Shulou.com)06/03 Report--

Example Flink cluster mvn package exec:java-Dexec.mainClass=org.apache.beam.examples.WordCount-Dexec.args= "--runner=FlinkRunner-- flinkMaster=-- filesToStage=target/word-count-beam-bundled-0.1.jar-- inputFile=/path/to/quickstart/pom.xml-- output=/tmp/counts"-Pflink-runner

Description of pipe parameters

FieldDescriptionDefault ValuerunnerThe pipeline runner to use. This option allows you to determine the pipeline runner at runtime.Set to FlinkRunnerto run using Flink.streamingWhether streaming mode is enabled or disabled; true if enabled. Set to true if running pipelines with unbounded PCollections.falseflinkMasterThe url of the Flink JobManager on which to execute pipelines. This can either be the address of a cluster JobManager, in the form "host:port" or one of the special Strings "[local]" or "[auto]". "[local]" will start a local Flink Cluster in the JVM while "[auto]" will let the system decide where to execute the pipeline based on the environment. [auto] filesToStageJar Files to send to all workers and put on the classpath. Here you have to put the fat jar that contains your program along with all dependencies.emptyparallelismThe degree of parallelism to be used when distributing operations onto workers.1checkpointingIntervalThe interval between consecutive checkpoints (I.E. Snapshots of the current pipeline state used for fault tolerance).-1L, I.E. DisablednumberOfExecutionRetriesSets the number of times that failed tasks are re-executed. A value of 0 effectively disables fault tolerance. A value of-1 indicates that the system default value (as defined in the configuration) should be used.-1executionRetryDelaySets the delay between executions. A value of-1 indicates that the default value should be used.-1stateBackendSets the state backend to use in streaming mode. The default is to read this setting from the Flink config.empty, i.e. Read from Flink config

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.

Views: 0

*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.

Share To

Internet Technology

Wechat

© 2024 shulou.com SLNews company. All rights reserved.

12
Report