Shulou(Shulou.com)05/31 Report--

This article is to share with you about what Apache Flink is. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

What is Apache Flink?

In the era of the surge in the amount of data, a large number of business data are generated in a variety of business scenarios. How to deal with these constantly generated data effectively has become a problem faced by most companies. With Yahoo's open source for hadoop, more and more big data processing technologies began to pour into people's eyes, such as the popular big data processing engine Apache Spark, which has basically replaced MapReduce to become the current big data processing standard. However, with the continuous growth of data and the continuous development of new technologies, people gradually realize the importance of real-time data processing. Compared with the traditional data processing mode, streaming data processing has higher processing efficiency and cost control ability. Flink is a distributed processing framework that can support high throughput, low latency and high performance among the technologies that have been developing in the open source community in recent years.

The evolution of data architecture

As shown in the figure, the biggest feature of the traditional single data architecture is centralized data storage, and most of the architecture is divided into computing layer and storage layer.

The initial efficiency of the single architecture is very high, but with the passage of time, more and more businesses, the system gradually becomes very large, more and more difficult to maintain and upgrade, the database is the only accurate data source, each application needs to access the database to obtain the corresponding data, if the database changes or problems, it will have an impact on the entire business system.

Later, with the emergence of micro-service architecture, enterprises began to adopt micro-service as the architecture of enterprise business system. The core idea of micro-service architecture is that an application is composed of several small, independent micro-services, which run in their own process and have no dependence on development and release. Different services can be built on different technical architectures according to different business needs, and can focus on limited business functions. As shown in the picture

Micro-service architecture

At first, the data warehouse was mainly built on the relational database. For example, Oracle, Mysql and other databases, but with the growth of enterprise data, relational databases have been unable to support the storage and analysis of large-scale data sets, because more and more enterprises begin to choose to build enterprise big data platform based on Hadoop. At the same time, it becomes simple and efficient to build different types of data applications on many Sql_on_hadhoop.

In the process of building an enterprise data warehouse, the data is often periodically synchronized from the business system to the big data platform, after completing a series of ETL conversion actions, the final formation of data Marts and other applications. However, for some time-demanding applications, such as real-time report statistics, there must be a very low delay to display statistical results, so the industry has proposed a set of Lambda architecture to deal with different types of data.

Big data lambada architecture

Big data platform includes Batch Layer for batch computing and Speed Layer for real-time computing. By integrating batch computing and stream computing in one platform, for example, batch computing is processed by Hadoop MapReduce and real-time data is processed by Apache Storm. This architecture solves the problems of different computing types to some extent, but the problem is that too many frameworks will lead to high platform complexity, high operation and maintenance costs and so on. It is also very difficult to manage the use of different types of computing frameworks in a resource management platform.

Later, with the emergence of Apache Spark's distributed memory processing framework, it is proposed to split the data into micro-batch processing mode for streaming data processing, so that batch computing and streaming computing can be completed in a set of computing framework. However, because Spark itself is based on batch mode, it can not deal with the native data flow perfectly and efficiently, so the support of convection computing is relatively weak. It can be said that the emergence of Spark is to some extent upgrade and optimize the Hadoop architecture.

Stateful flow computing architecture

The nature of data generation is actually a series of real events. The different architectures mentioned above actually violate this nature to a certain extent. It is necessary to process business data with a certain delay, and then get accurate results based on business data statistics. In fact, due to the limitations of streaming computing technology, it is difficult for us to calculate and directly produce statistical results in the process of data generation, because it not only has very high requirements for the system, but also has to meet many goals such as high performance, high throughput, low delay and so on.

The biggest advantage of stateful computing is that there is no need to re-take the original data from external storage for full computing, because the cost of this computing method can be very high.

Flink implements a real-time streaming computing framework with high throughput, low latency and high performance by implementing the Google Dataflow streaming computing model. At the same time, Flink supports highly fault-tolerant state management to prevent the state from being lost due to system anomalies in the process of computing. Flink periodically maintains the state through the distributed snapshot technology Checkpoints, so that the correct results can be calculated even in the case of system downtime or abnormal.

The specific advantages of Flink are as follows:

At the same time, supporting high throughput, low latency and high performance Flink is the only distributed streaming data processing framework that integrates high throughput, low latency and high performance in the open source community. For example, Apache Spark can only take into account high throughput and high performance characteristics, mainly because it can not achieve low latency guarantee in Spark Streaming streaming computing, while streaming computing framework Apache Storm can only support low latency and high performance characteristics, but can not meet the requirements of high throughput. It is very important for the distributed streaming computing framework to meet the three goals of high throughput, low latency and high performance.

Supporting the concept of event time (Event Time) plays an important role in the field of streaming computing, but at present, most frame window computing uses system time (Process Time), which is also the current time of the system host when the event is transmitted to the computing framework for processing. Flink can support window computing based on event time (Event Time) semantics, that is, using the time generated by events. This event-driven mechanism enables the streaming system to calculate accurate results even if events arrive out of order, maintaining the timing of the original events and avoiding the impact of network transmission or hardware systems as much as possible.

Flink supports stateful computing, which implements state management in version 1.4. The so-called state means that the intermediate result data of the operator is saved in memory or in the file system in the process of streaming computing. After the next event enters the operator, the current result can be calculated from the intermediate result from the previous state, so that there is no need to count the results based on all the original data each time. This method greatly improves the performance of the system and reduces the resource consumption of the data computing process. For streaming computing scenarios with large amount of data and very complex operation logic, stateful computing plays a very important role.

Support for highly flexible windows operations

In streaming applications, the data is continuous, so it is necessary to stream the data through a window for a certain range of aggregate calculations, such as counting how many users have clicked on a web page in the past minute. In this case, we must define a window to collect data within the last minute and recalculate the data in this window. Flink divides windows into window operations based on Time, Count, Session, and Data-driven. Windows can be customized with flexible trigger conditions to support complex streaming modes, and users can define different window trigger mechanisms to meet different needs.

Fault-tolerant Flink based on lightweight distributed snapshot (Snapshot) can run on thousands of nodes, disassemble the flow of a large computing task into small computing processes, and then distribute the tesk to parallel nodes for processing. In the process of task execution, it can automatically find the problems of data inconsistency caused by errors in event processing, such as node downtime, network transmission problems, or the restart of computing services due to user upgrades or repair problems. In these cases, through the Checkpoints based on distributed snapshot technology, the state information in the execution process is persisted. Once the task stops abnormally, Flink can automatically recover the task from the Checkpoints to ensure the consistency of the data in the process of processing.

The implementation of independent memory management based on JVM is an important part of all computing frameworks, especially for computing scenarios with a large amount of computation, how to manage data in memory is very important. For memory management, Flink implements its own memory management mechanism to minimize the impact of JVM GC on the system. In addition, Flink converts all data objects into binaries and stores them in memory through serialization / deserialization, reducing the size of data storage and reducing the use of memory space more effectively, reducing the risk of performance degradation or task anomalies brought about by GC. Therefore, Flink is more stable than other distributed processing frameworks and will not affect the operation of the entire application because of JVM GC and other problems.

Save Points (SavePoint) for streaming applications running around the clock, data is accessed continuously. The termination of the application within a period of time may lead to data loss or inaccurate calculation results, such as upgrading the cluster version, downtime operation and maintenance operations, and so on. It is worth mentioning that Flink saves the snapshot of the task execution on the storage medium through Save Points technology, and when the task is restarted, it can directly engage in the saved Save Points to restore the original computing state, so that the task continues to run according to the state before downtime. Save Points technology allows users to better manage and operate real-time streaming applications.

Thank you for reading! This is the end of this article on "what is Apache Flink?". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.