Shulou(Shulou.com)05/31 Report--

How to migrate Flink tasks to real-time computing, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain for you in detail, people with this need can come to learn, I hope you can gain something.

Usually, users mainly use Flink run offline, which can cause some problems, such as: the same configuration changes due to version; unable to quickly switch between different versions; unable to recover from previous jobs.

So how do you migrate from offline to online? How to introduce it to you through the following four parts.

How to migrate: from Flink run to Stream Computing platform

How the offline commands in the following figure are migrated to online. First, open VVP UI and add the basic configuration, including: job name, Jar URI, some major parameters and parallelism. You can also click Advanced configuration to configure more information.

For example, the behavior configuration in an advanced configuration is divided into three parts: upgrade strategy, initial state, and recovery strategy. Generally, the upgrade strategy is Stateless, the initial state is Running, and the recovery strategy is Latest State.

Select Stateless for the upgrade strategy, which means that after editing and saving the configuration for the already running job, the original job will be stopped directly, and then an updated job will be restarted; if you select Stateble, if you edit and save the configuration for the running job, the job will first do a Savepoint, then load the new configuration and use the Savepoint to start a new job.

LatestSavepoint is selected as the recovery strategy, which means that if the job is paused, if it is started again, it will start with the latest Savepoint.

The above is an important part of the Flink configuration, and the following image shows another part of the configuration where you can set the Checkpoint interval, etc., followed by resource configuration. In the log configuration part, you can choose the location to save the log, so that if there is a problem with the job in the future, you can easily troubleshoot the problem.

After the Flink job is configured and started to run, if an exception occurs, you can view the running status and problems through the run event. You can also open the Job Manager of Flink UI to view the log.

2. How to add other dependent Jar in the case of multiple Jar

Some users have custom dependencies and cannot solve the problem through fat jar. For example, A users have their own main Jar and some other Jar, and the application scenarios of these Jar are different, so you need to upload the Jar package on the resource management page first, and then you can use the file on the page after the upload is completed.

After uploading, go to the advanced configuration, find the "extra dependent Jar", drop down and select the Jar package you just uploaded.

Third, how do users define jobs through configuration files

Upload files in resource management, and then select add dependencies in the advanced configuration to select the desired dependencies. If main class needs to read a file in the startup function, first add the file through dependency, and then read the file according to the prompt.

The above two ways are the way to place the user's Jar, where jar and other files can be stored.

4. How to reuse the original state to accelerate job recovery

If you find a problem while running a job, you usually stop and find the problem and then restart it. The user needs to specify the OSS required by Checkpoint at the time of creation. If Checkpoint is enabled, subsequent jobs can be restored directly from Checkpoint.

First confirm that the recovery strategy in the advanced configuration is LatestState or LatestSavepoint. Under this premise, you can pause the job (never stop the job), pause will trigger Savepoint, and when you click start again, it will resume from this Savepoint point.

Open Flink UI, and then click the job snapshot. In the job snapshot, you can view the Flink Checkpoint metric, which displays information such as the number of Restore and the address of the last Restore, to determine whether the job is restored from the latest State.

When the job is running and you want to make a copy of the current job, how to do it? Click Checkpoint on the job control page, find the historical snapshot and click the corresponding rightmost "copy Job from this Snapshot". Then the operation of copying the current job is completed, including the Jar package configuration of the running job. After clicking start, you can see from the job snapshot that the new job was started from the most recent Savepoint of the previous job.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.