Shulou(Shulou.com)05/31 Report--

This article mainly shows you "what are the basic components of flink", the content is easy to understand, clear, hope to help you solve your doubts, the following let the editor lead you to study and learn "what are the basic components of flink" this article.

Flink, an open source computing platform for distributed data stream processing and batch data processing, can support streaming and batch processing applications based on the same Flink runtime. It has similar functions to mapreduce,storm and spark.

First of all, let's take a look at the general picture.

Let's talk about the specific components.

JobManager,taskManager

JobManager:JobManager is the coordinator of the Flink system, which is responsible for receiving the Flink Job and scheduling the execution of multiple Task that make up the Job. At the same time, JobManager is also responsible for collecting the status information of the Job and managing the slave node TaskManager in the Flink cluster.

JobManager

JobManager is the coordinator of the Flink system and an independent actor, which is responsible for receiving the Flink Job and scheduling the execution of multiple Task that make up the Job. At the same time, JobManager is also responsible for collecting the status information of the Job and managing the slave node TaskManager in the Flink cluster. The management functions that JobManager is responsible for, the events it receives and handles mainly include:

RegisterTaskManager

When the Flink cluster starts, TaskManager registers with JobManager, and if the registration is successful, JobManager replies to TaskManager with the message AcknowledgeRegistration.

SubmitJob

Within the Flink program, the Flink Job is submitted to JobManager through Client, where the basic information of Job is described in the form of JobGraph in the message SubmitJob.

CancelJob

Request to cancel the execution of a Flink Job. The CancelJob message contains the ID of the Job. If it succeeds, it returns the message CancellationSuccess. If it fails, it returns the message CancellationFailure.

UpdateTaskExecutionState

TaskManager requests JobManager to update the status information of ExecutionVertex in ExecutionGraph, and returns true if the update is successful.

RequestNextInputSplit

The Task running on TaskManager requests the next input Split to be processed, and returns NextInputSplit if it succeeds.

JobStatusChanged

ExecutionGraph sends this message to JobManager to indicate a change in the state of Flink Job, such as RUNNING, CANCELING, FINISHED, and so on.

TaskManager

The TaskManager is also an Actor, which is the Worker that is actually responsible for performing the calculation, and a set of Task on which the Flink Job is executed. Each TaskManager is responsible for managing the resource information on its node, such as memory, disk, and network, and reports the status of the resource to the JobManager at startup. The TaskManager side can be divided into two phases:

Registration stage

TaskManager registers with JobManager, sends RegisterTaskManager messages, waits for JobManager to return AcknowledgeRegistration, and then TaskManager can initialize the process.

Operable stage

At this stage, TaskManager can receive and process messages related to Task, such as SubmitTask, CancelTask, FailTask. If TaskManager cannot connect to JobManager, TaskManager loses contact with JobManager and automatically enters the "registration phase". Only when registration is completed can Task-related messages continue to be processed.

Client

When a user submits a Flink program, a Client will be created first. The Client will first preprocess the Flink program submitted by the user and submit it to the Flink cluster, so Client needs to obtain the address of JobManager from the Flink program configuration submitted by the user, establish a connection to JobManager, and submit the Flink Job to JobManager. Client assembles a JobGraph of the Flink program submitted by the user and submits it in the form of JobGraph. A JobGraph is a Flink Dataflow, which is a DAG made up of multiple JobVertex. A JobGraph contains the following information for a Flink program: JobID, Job name, configuration information, a set of JobVertex, and so on.

These are all the contents of the article "what are the basic components of flink?" Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.