Shulou(Shulou.com)06/01 Report--

This article to share with you is about blink to solve a flink analysis pain point example analysis, Xiaobian think quite practical, so share to everyone to learn, I hope you can read this article after some harvest, not much to say, follow Xiaobian to see it.

Today is also mainly a test and research blink. As mentioned above, for the big data framework, in addition to understanding the underlying principles, but also to master how to analyze the performance bottleneck, blink in addition to optimizing from the bottom, but also optimize the web ui of flink, and there are relatively large improvements.

For the official description of blink optimization points, you can check the readme.md documentation in flink's github blink branch.

https://github.com/apache/flink/tree/blink

First of all, import blinks into ideas.

idea directly supports importing blinks via git

The window below pops up and you enter the address of flink in the url field.

And then wait,

After successful import, switch to blink in the lower right corner

Then you can compile. Before compiling, you need to comment out the content of code style detection in pom.xml. then perform

mvn clean package -DskipTests

After compilation, the installation file of blink will be generated in the build-target directory, and then you can copy it out. If you just want to test in local mode, you can directly execute it.

bin/start-cluster.sh

After the startup is complete, enter host:8081 in the browser to log in to the web ui of blink.

You can see that the content style of the web ui has been completely restructured, and the details are very rich.

Here are the official web optimization points:

We have improved the usability and performance of Flink Runtime Web, adding a lot of new features from resource usage, job tuning, log query and other dimensions, making it easier for users to operate Flink jobs.

Resource Usage: Add Cluster, TaskManager and Job three levels of resource information, so that the application and usage of resources can be clearly seen.

Job tuning: Topological relationships and data flow of jobs can be traced back to the Operator level. Vertex adds InQueue, OutQueue and other indicators to track data backpressure, filtering and tilt conveniently.

Log query: The log function of TaskManager and JobManager has been greatly enhanced. Multiple dimensions such as Job, Vertex and SubTask can be associated with the corresponding log. Multiple log file access entries are provided, as well as pagination display query and log highlighting functions.

Interaction optimization: The page interaction logic is optimized as a whole, and most of the related information can be queried and compared in a single page, reducing a lot of unnecessary jumps.

Performance improvement: Angular 7.0 was used for overall refactoring, and page performance was more than doubled. In the case of large amounts of data will not occur page death or stuck situation.

log management

Here, the main thing I want to mention is about log-related monitoring. We all know that flink logs only have taskmanager and jobmanager. If the cluster runs multitasking, it will cause log analysis to be very painful, so the current situation on yarn is a single task.

blink log, give us a breakdown of the file, so that we can view it for the application

Run two simple tasks

bin/flink run examples/streaming/

bin/flink run examples/streaming/Kafka010Example.jar --input-topic test --output-topic boo --bootstrap.servers mt-mdh.local:9093 --groupid test

Check the taskmanager log, you can see that TaskManager and JobManager log function has been greatly enhanced, from Job, Vertex, SubTask and other dimensions can be associated with the corresponding log, provide multi-log file access entry, and pagination display query and log highlight function.

Of course, the log file name discrimination is not very high, mainly according to the host of the submission task, and then submit the order to find: the corresponding relationship between the log and the program, but has greatly improved our ability to analyze the log, because the original flink log is more concentrated in taskmanager, then if the running time is long, the log file will roll larger and larger, and the running task will cross the log, and then cause the problem analysis to become more difficult.

The above is an example analysis of blink to solve a flink analysis pain point. Xiaobian believes that some knowledge points may be seen or used in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.