Shulou(Shulou.com)06/01 Report--

What is the concept and characteristics of Flink? in view of this question, this article introduces the corresponding analysis and solutions in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible way.

Flink concept

ApacheFlink is an open source computing platform for distributed data stream processing and batch data processing. It can support streaming and batch processing applications based on the same Flink runtime.

The existing open source computing solutions will regard streaming and batch processing as two different application types, because the SLA (Service-Level-Aggreement) they provide is completely different: stream processing generally needs to support low latency and Exactly-once guarantee, while batch processing needs to support high throughput and efficient processing.

Flink looks at stream processing and batch processing from another perspective, unifying the two: Flink fully supports stream processing, that is, the input data stream is unbounded when viewed as a stream processing; batch processing is treated as a special stream processing, but its input data stream is defined as bounded.

Characteristics of Flink

Apache Flink is an open source computing platform for distributed data stream processing and batch data processing. It can support streaming and batch processing applications based on the same Flink runtime.

The existing open source computing solutions will regard streaming and batch processing as two different application types, because the SLA (Service-Level-Aggreement) they provide is completely different: stream processing generally needs to support low latency and Exactly-once guarantee, while batch processing needs to support high throughput and efficient processing.

Flink looks at stream processing and batch processing from another perspective, unifying the two: Flink fully supports stream processing, that is, the input data stream is unbounded when viewed as a stream processing; batch processing is treated as a special stream processing, but its input data stream is defined as bounded.

Flink technology stack

Flink first supports the execution environment of Local, and all module components can run in the same JVM process, mainly to facilitate development and debugging. When developing Flink applications, users can run locally or set breakpoints to debug the code in the IDE environment. In addition, Flink supports Standalone mode for distributed deployment. JobManager and TaskManager of Flink can be deployed on multiple nodes to form a cluster, manage cluster resources, and execute distributed tasks. At the same time, Flink can also entrust YARN to manage resources, and Job of Flink applies for resources through YARN. Finally, Flink also supports Google and Amazon's public cloud platforms, and Flink's Job can be submitted directly to the public cloud for execution. The same Flink business logic developed by users can be migrated to different execution environments at any time without any modification.

Flink architecture

Client is responsible for submitting Flink jobs. First, the user's Flink Job is translated and optimized into a graphical Dataflow, and then submitted to JobManager,JobManager to split the Flink DataFlow into distributed task. The task is scheduled according to the resource configuration of the cluster, and the task is distributed to the TaskManger for execution. There may be more than one TaskManager that executes a specific task and is responsible for data transfer between task and task. TaskManger, which is different from MapReduce,Flink, is similar to Spark's Executor,task that executes in TaskManager at the granularity of threads rather than processes. In Standalone mode, JobManger and TaskManager are responsible for resource management, while in YARN mode, JobManager is equivalent to ApplicationMaster, and TaskManger is equivalent to Executor. Only responsible for the scheduling and execution of task.

Flink programming model

The lowest level of abstraction only provides a stateful flow. In fact, most applications do not need the underlying abstraction mentioned above, but are programmed against the core API (Core APIs). For example, DataStream API (bounded or unbounded stream data) and DataSet API (bounded dataset) Table API are table-centric declarative DSL, where tables may change dynamically (when expressing stream data).

Flink program execution process

Client is responsible for submitting Flink jobs. First, the user's Flink Job is translated and optimized into a graphical Dataflow, and then submitted to JobManager,JobManager to split the Flink DataFlow into distributed task. The task is scheduled according to the resource configuration of the cluster, and the task is distributed to the TaskManger for execution. There may be more than one TaskManager that executes a specific task and is responsible for data transfer between task and task. TaskManger, which is different from MapReduce,Flink, is similar to Spark's Executor,task that executes in TaskManager at the granularity of threads rather than processes. In Standalone mode, JobManger and TaskManager are responsible for resource management, while in YARN mode, JobManager is equivalent to ApplicationMaster, and TaskManger is equivalent to Executor. Only responsible for the scheduling and execution of task

The answer to the question about the concept and characteristics of Flink is shared here. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel to learn more about it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.