Shulou(Shulou.com)05/31 Report--

This article will explain in detail what the Flink architecture is like. The editor thinks it is very practical, so I share it for you as a reference. I hope you can get something after reading this article.

1. Basic component stack

Friends who know Spark will find that the architecture of Flink is very similar to that of Spark. In the whole software architecture, it also follows the concept of hierarchical architecture design, which not only reduces the system coupling, but also provides rich and friendly interfaces for upper-level users to build Flink applications.

Flink is divided into three layers: API&Libraries layer, Runtime core layer and physical deployment layer from top to bottom.

API&Libraries layer

As a distributed data processing framework, Flink also provides interfaces to support computing and batch computing, and abstracts component libraries of different application types, such as stream-based CEP (complex event processing library), SQL&Table library and batch-based FlinkML (machine learning library), Gelly (graph processing library) and so on. The API layer includes DataStream API for building stream computing applications and DataSet API for batch computing applications, both of which provide users with rich high-level API for data processing, such as Map and FlatMap operations, as well as relatively low-level Process Function API, where users can directly manipulate underlying data such as status and time.

Runtime core layer

This layer is mainly responsible for providing basic services for different interfaces in the upper layer, and is also the core implementation layer of the Flink distributed computing framework, supporting the execution of distributed Stream jobs, mapping from JobGraph to ExecutionGraph, task scheduling and so on. The DataSteam and DataSet are transformed into a unified executable Task Operator to deal with batch computing and streaming computing at the same time under the streaming engine.

Physical deployment layer

This layer mainly involves the deployment mode of Flink. Currently, Flink supports a variety of deployment modes: local, cluster (Standalone, YARN), cloud (GCE/EC2), Kubenetes. Flink can support the deployment of different platforms through this layer, and users can choose to use the corresponding deployment model according to their needs.

2. Basic architecture diagram

Flink basic architecture diagram

Flink system is mainly composed of two components, which are JobManager and TaskManager,Flink architecture and follow the design principles of Master-Slave architecture. JobManager is a Master node and TaskManager is a Worker (Slave) node. All components communicate with each other through Akka Framework, including information such as the status of the task and Checkpoint triggers.

1.Client client

The client is responsible for submitting the task to the cluster, building an Akka connection with JobManager, then submitting the task to JobManager, and interacting with JobManager to obtain the task execution status. The client submission task can be submitted by CLI or by using Flink WebUI, or you can specify the RPC network port of JobManager in the application to build ExecutionEnvironment submission Flink application.

2.JobManager

JobManager is responsible for the task scheduling and resource management of the entire Flink cluster, obtains the submitted application from the client, and then allocates the corresponding TaskSlots resources to the submitted application according to the use of TaskSlot on the TaskManager in the cluster and commands TaskManager to start the application obtained from the client. JobManager is equivalent to the Master node of the whole cluster, and there is only one active JobManager in the whole cluster, which is responsible for the task management and resource management of the whole cluster. JobManager and TaskManager communicate with each other through Actor System to obtain the execution of the task and send the execution of the application to the client through Actor System. At the same time, during the execution of the task, Flink JobManager will trigger the Checkpoints operation. After each TaskManager node receives the Checkpoint trigger instruction, it completes the Checkpoint operation. All the Checkpoint coordination processes are completed in Flink JobManager. When the task is completed, the Flink will feed back the information of the task execution to the client, and release the resources in the TaskManager for the next task submission.

3.TaskManager

TaskManager is equivalent to the Slave node of the whole cluster, which is responsible for the execution of specific tasks and the resource application and management of the corresponding tasks on each node. The client compiles and packages the written Flink application and submits it to JobManager, and then JobManager assigns the task to the TaskManager node with resources according to the resources already registered in JobManager, and then starts and runs the task. TaskManager receives the tasks that need to be deployed from JobManager, and then uses Slot resources to start Task, establish a network connection for data access, receive data, and start data processing. At the same time, the data exchange between TaskManager is carried out through the way of data flow.

It can be seen that the task of Flink runs in a multi-threaded way, which is very different from the way of MapReduce multi-JVM process. Flink can greatly improve the efficiency of CPU use, share system resources through TaskSlot between multiple tasks and Task, and manage multiple TaskSlot resource pools in each TaskManager to manage resources effectively.

This is the end of this article on "what the Flink architecture is like". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.