Shulou(Shulou.com)06/02 Report--

This article will explain in detail what is the secret about the smooth migration of Storm tasks to Flink. The content of the article is of high quality, so Xiaobian shares it with you for reference. I hope you have a certain understanding of relevant knowledge after reading this article.

Flink-Storm is an official beta tool provided by Flink for Flink-compatible Storm programs, and the related code has been removed since Release 1.8. It mainly describes how 58 real-time computing platform optimizes Flink-Storm and realizes smooth migration of large-scale Storm tasks in real scenes based on Flink-Storm.

background

58 Real-time computing platform is designed to provide stable and efficient real-time computing services for business departments of the Group. It is mainly built on Storm and Spark Streaming, but it also faces some problems in the use process, mainly including Storm's insufficient throughput and operation and maintenance problems caused by multiple clusters. Spark Streaming cannot meet the requirements of low latency. After Apache Flink was open sourced, its advantages in architectural design, computational performance, and stability led us to adopt Flink as the computational engine for the next generation of real-time computing platforms. At the same time, Wstream, a one-stop high-performance real-time computing platform based on Flink, has been developed to support Flink jar, Stream Sql, Flink-Storm and other diversified task construction methods.

While improving the Flink platform, we also launched the Storm Task Migration Flink Project, which aims to improve the overall efficiency of the real-time computing platform and reduce machine costs and operation and maintenance costs.

Storm vs Flink

Although Flink works well with Storm as a high-performance compute engine, we still encountered some issues during the business migration process:

1 . user learning costs for Flink;

Re-development based on Flink is labour-intensive;

Stream-SQL can satisfy rapid development, reduce learning costs and development workload, but it cannot satisfy some complex scenarios.

Therefore, we decided to use Flink-Storm officially provided by Flink for migration, which ensures the stability of migration without requiring users to modify Storm code logic.

Flink-Storm principle

Storm topology is constructed by Storm Native Topology Builder.

FlinkTopology.createTopology(builder) converts StormTopology to Streaming Dataflow corresponding to Flink.

SpoutWrapper is used to convert spout into RichParallelSourceFunction, and OutputFields of spout into TypeInformation of source.

BoltWrapper is used to convert a bolt to a corresponding operator, where grouping is converted to the corresponding operation on the DataStream of spouts.

After building Flink Topology, you can generate StreamGraph through StreamExecutionEnvironment to obtain JobGraph, and then submit JobGraph to Flink runtime environment.

practice

Flink-Storm as the official Flink-compatible Storm program provides us with the feasibility of seamless migration, but as a beta version, there are many situations in the actual use process that cannot meet the real situation, mainly including version, functional bugs, complex logic compatibility, and inability to support yarn. The following will be mainly divided into platform level and user level to describe our use and improvement.

Platform level 1. version

Apache Flink version 1.6 is currently online, the Flink-Storm module is based on Storm 1.0, and our platform runs Storm versions 0.9.5 and 1.2.

1.1 For Storm 1.2 run tasks, the Storm 1.0 API is fully compatible with version 1.2, so simply switch storm-core, which the Flink-Storm module relies on, to 1.2.

1.2 For Storm 0.9.5 task, since Storm 1.0 API is not compatible with 0.9.5, you need to modify the dependency storm-core to 0.9.5, and modify all Storm-related APIs in Flink-Storm module, mainly switching package path.

1.3 rebuild flink-storm package mvn clean package -Dmaven.test.skip=true -Dcheckstyle.skip=true

2. function

2.1 transitive semantic guarantee

Storm uses ACK mechanism to achieve transitive semantic assurance, we did not port Storm's ACK mechanism to Flink-Storm. As a result, certain functions that rely on ACK mechanisms are limited. For example, Kafka spout stores the consumption state in ZK, and the state update needs to rely on the ACK mechanism. After the tuple tree ends, spout will trigger the state update, indicating that the message has been completely processed, thus achieving the delivery guarantee at least once. Storm also provides support for at most once, spouts sending messages that trigger status updates directly without waiting for the tuple tree to end. We use Storm's implementation at most once, on the basis of Kafka spout's implementation at most once, by implementing Flink Checkpoint's state mechanism, we achieve Flink-Storm's task at least once. Storm tasks migrate to Flink, delivery guaranteed unchanged.

2.2 tick tuple mechanism

Storm uses tick tuple mechanism to realize timing function, and functions such as message timeout retransmission and Bolt timing trigger all depend on tick tuple mechanism. Storm version 0.9.5 does not implement the window function, users can use the tick tuple mechanism to simply implement the window function. We have also added tick tuple support for Flink-Storm, which is used in the same way as Storm, configuring topology.tick.tuple. freq.secs parameter, that is, the tick tuple function is enabled.

2.3 All-grouping support under multi-input

AllGrouping corresponds to Flink is Broadcast. As shown in the figure, bolt-1 has two inputs. In this case, the implementation of the original flink-storm, spout-2 to bolt-1 data partition behaves like Rebalance (Flink terminology), not Broadcast. We optimized this scenario so that the data groupings behave the same as in Storm.

3.Runtime

Flink-Storm supports local and standalone mode task submission by default, and cannot submit tasks to yarn cluster. We chose yarn mode at the beginning of building Flink cluster, which is convenient for cluster resource management and unified real-time computing platform. Therefore, we need to implement runtime function supporting yarn by ourselves, which mainly involves yarn client design.

YARN Client Implementation Mechanism

The whole module is mainly divided into four parts, in which client is used to call Flink-Storm program transformation interface to get Flink jobGraph. Configuration parameters are used to initialize Flink and yarn-related configurations, build runtime environments, and command-line tools are mainly used for more flexible management. YarnClient mainly implements ApplicationClientProtocol interface, completes interaction with ResourceManager and ApplicationMaster, and implements Flink job submission and monitoring.

4. Mission deployment

To facilitate task submission and integration into the Wstream platform, Flink-like command-line submission is provided:

User level 1.maven dependencies

The platform uploads the compiled package to the company maven private service for users to download the corresponding version of Flink-Storm dependency package:

2. code churn

Users need to change the way Storm submits tasks to Flink submission, and nothing else needs to be changed.

summary

Through the optimization and use of Fink-Storm, we have successfully completed the migration and offline of multiple Storm cluster tasks, saving computing resources by more than 40% on the basis of ensuring real-time performance and throughput. At the same time, yarn can be used to manage the real-time computing platform without maintaining multiple Storm clusters, which improves the overall utilization rate of platform resources and reduces the workload of platform operation and maintenance.

What is the secret about the smooth migration of Storm tasks to Flink? I hope the above content can be helpful to everyone and learn more. If you think the article is good, you can share it so that more people can see it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.