Shulou(Shulou.com)06/03 Report--

Author: Wang Feng

Finishing: Han Fei

This article is mainly sorted out from the speech delivered by Wang Feng (nickname: Mo Wen), a senior technical expert of Alibaba Computing platform Division, at the developer Ecology Summit of Yunqi Conference.

Cdn.xitu.io/2019/5/5/16a8607970f49bd7?w=2874&h=1916&f=jpeg&s=283674 ">

With the massive growth of data, the future of the digital age is coming. Whether it is the small data era of knot recording, or the era of big data that we are experiencing, the boundary of computing is being infinitely broadened, and the value of data is no longer difficult to be calculated. Nowadays, when it comes to big data, we have to mention the hottest next-generation big data computing engine Apache Flink (hereinafter referred to as Flink). This article will combine the past life and this life of Flink, from a business point of view, to explain to you: why Ali chose Flink?

The wood that embraces each other is born in the end.

With the advent of the era of artificial intelligence and the outbreak of the amount of data, the most common approach of data business in the typical business scenario of big data is to select batch processing technology to deal with all data, and to use streaming computing to process real-time incremental data. In most business scenarios, the user's business logic is often the same in batch and flow processing. However, the two sets of computing engines that users use for batch and stream processing are different. As a result, users usually need to write two sets of code. There is no doubt that this brings some additional burden and cost. Alibaba's commodity data processing often needs to face two different business process problems of increment and full volume, so Ali is wondering whether we can have a unified set of big data engine technology. Users only need to develop a set of code according to their own business logic. In this way, in a variety of different scenarios, whether it is full data or incremental data, or real-time processing, a set of solutions can all be supported, which is the background and original intention of Ali choosing Flink.

At present, there are many options for open source big data computing engine, such as Storm, Samza, Flink, Kafka Stream, etc., batch processing such as Spark, Hive, Pig, Flink and so on. For computing engines that support both streaming and batch processing, there are only two options: one is Apache Spark and the other is Apache Flink.

From the technical, ecological and other aspects of comprehensive consideration, first of all, the technical concept of Spark is based on batch to simulate flow computing. Flink, by contrast, simulates batch computing based on stream computing.

From the perspective of technological development, using batches to simulate flow has some technical limitations, and this limitation may be difficult to break through. Flink, on the other hand, simulates batches based on streams and is more scalable in technology. In the long run, Ali decided to use Flink to make a unified, general-purpose big data engine as a future selection.

Flink is a low-latency, high-throughput, unified big data computing engine. In Alibaba's production environment, Flink's computing platform can process hundreds of millions of messages or events per second with millisecond latency. At the same time, Flink provides a consistent semantics of Exactly-once. The correctness of the data is guaranteed. This enables the Flink big data engine to provide financial-level data processing capabilities.

The status quo of Flink in Ali

The platform based on Apache Flink in Alibaba was officially launched in 2016, and started with Alibaba's search and recommendation. At present, all the businesses of Alibaba, including all subsidiaries of Alibaba, have adopted a real-time computing platform based on Flink. At the same time, Flink computing platform runs on the open source Hadoop cluster. YARN of Hadoop is used as resource management and scheduling, and HDFS is used as data storage. Therefore, Flink can be seamlessly docked with the open source big data software Hadoop.

At present, this Flink-based real-time computing platform not only serves within Alibaba Group, but also provides Flink-based cloud product support to developers through Aliyun's cloud product API.

How is the large-scale application of Flink in Alibaba?

Scale: scale is an important indicator of whether a system is mature or not. Alibaba initially launched Flink with only a few hundred servers, but now the scale has reached tens of thousands, and such scale is only a handful in the world.

State data: based on Flink, the internal accumulated state data is already at the PB level.

Events: today, more than a trillion pieces of data are processed on Flink's computing platform every day.

TPS: it can handle more than 472 million visits per second during the peak period. The most typical application scenario is Alibaba's double 11 big screen.

The Development Road of Flink

Next, from the perspective of open source technology, let's talk about how Apache Flink was born and how it grew. And how did Ali enter at this critical point of growth? And what contributions and support have been made to it?

Flink was born in StratoSphere, a research project of big data in Europe. The project is a research project of the Technical University of Berlin. In the early days, Flink did Batch computing, but in 2014, the core members of StratoSphere hatched Flink, donated Flink to Apache in the same year, and later became Apache's top big data project. At the same time, the mainstream direction of Flink computing was positioned as Streaming, that is, streaming computing is used to do all big data's calculations, which is the background of the birth of Flink technology.

In 2014, as the big data engine that focuses on streaming computing, Flink began to emerge in the open source big data industry. Different from Storm, Spark Streaming and other streaming computing engines, it is not only a high-throughput, low-latency computing engine, but also provides many advanced functions. For example, it provides stateful computing, supports state management, supports highly consistent data semantics, and supports Event Time,WaterMark to deal with message disorder.

Core concepts and basic concepts of Flink

What distinguishes Flink from other stream computing engines is actually state management.

What is status? For example, to develop a stream computing system or task to do data processing, you may often have to count the data, such as Sum, Count, Min, Max, these values need to be stored. Because it is constantly updated, these values or variables can be understood as a state. If the data source is reading Kafka, RocketMQ, you may want to record where to read, and record Offset, these Offset variables are the states to be calculated.

Flink provides built-in state management so that these states can be stored inside the Flink without having to store it on an external system. The advantage of this is that, first, it reduces the dependence and deployment of the computing engine on external systems, making the operation and maintenance easier; second, it brings a great improvement in performance: if you access it from the outside, such as Redis,HBase, it must be through the network and RPC. If accessed from within Flink, it accesses these variables only through its own process. At the same time, Flink periodically persists these states to Checkpoint, storing Checkpoint in a distributed persistence system, such as HDFS. In this way, when there is any failure in Flink's task, it recovers the state of the entire flow from the most recent Checkpoint and then continues to run its flow processing. There is no data impact on the user.

How does Flink achieve no data loss and data redundancy in the process of Checkpoint recovery? To ensure accurate calculation?

The reason for this is that Flink uses a set of very classical Chandy-Lamport algorithm, and its core idea is to regard the flow calculation as a streaming topology, regularly insert a special Barriers from the head source point of the topology, and broadcast the Barriers to the downstream from the upstream. Each node receives all the Barriers and will Snapshot the State once. When each node has done the Snapshot, the whole topology will complete the Checkpoint once. Next, whatever failure occurs, it recovers from the most recent Checkpoint.

Flink uses this classical algorithm to ensure strong consistency in semantics. This is also the core difference between Flink and other stateless flow computing engines.

Here's how Flink solves the disorder problem. For example, the playing order of Star Wars, if you watch it according to the release time, you may find the story jumping.

In flow computation, it is very similar to this example. The time of arrival of all messages is inconsistent with the time at which it actually occurs at the source, the online system Log. In stream processing, it is hoped that the message will be processed in the order in which the message actually occurs at the source, rather than the time it really arrives in the program. Flink provides some advanced technologies of Event Time and WaterMark to solve the problem of disorder. Enables the user to process the message in an orderly manner. This is a very important feature of Flink.

Next, we will introduce the core concepts and core concepts at the start of Flink, which is the first stage of Flink development; the second stage is 2015 and 2017, which is also the time for Flink development and Alibaba's involvement. The story begins in mid-2015, when we conducted a survey in the search business department. At that time, Ali had its own batch processing technology and stream computing technology, both self-developed and open source. However, in order to think about the direction and future trend of the next generation big data engine, we have done a lot of research on new technologies.

Combined with a large number of research results, we finally come to the conclusion that to solve the computing needs of GM big data, the computing engine of batch-stream integration is the development direction of big data technology, and finally we chose Flink.

But the 2015 Flink is not mature enough, neither scale nor stability has yet to be put into practice. Finally, we decided to set up a Flink branch within Ali and make a lot of modifications and improvements to Flink to adapt it to the super-large-scale business scenario like Alibaba. In this process, our team has not only made a lot of improvements and optimizations in the performance and stability of Flink, but also made a lot of innovations and improvements in the core architecture and functions, and contributed them to the community, such as Flink's new distributed architecture, incremental Checkpoint mechanism, Credit-based-based network flow control mechanism and Streaming SQL.

Alibaba's contribution to the Flink Community

We give two design cases. The first is that Alibaba reconstructs the distributed architecture of Flink and makes a clear layering and decoupling of Flink Job scheduling and resource management. The first benefit of this is that Flink can run natively on a variety of open source resource managers. With the improvement of this distributed architecture, Flink can run natively on the two most common resource management systems, Hadoop Yarn and Kubernetes. At the same time, the task scheduling of Flink is changed from centralized scheduling to distributed scheduling, so that Flink can support larger clusters and better resource isolation.

The other is to implement the incremental Checkpoint mechanism, because Flink provides stateful computing and periodic Checkpoint mechanism. If there is more and more internal data and keep doing Checkpoint, the Checkpoint will become bigger and bigger, which may eventually lead to inability to do it. When incremental Checkpoint is provided, Flink automatically discovers which data is incremental and which data has been modified. At the same time, only these modified data are persisted. In this way, Checkpoint will not become more and more difficult to do over time, and the performance of the whole system will be very smooth, which is a very important feature that we contribute to the community.

After improving the capacity of Flink Streaming from 2015 to 2017, the Flink community has gradually matured. Flink has also become the most mainstream computing engine in the Streaming field. Because Flink wanted to do a unified big data engine in 2018, in order to achieve this goal, Alibaba proposed a new unified API architecture and a unified SQL solution. At the same time, after the various functions of stream computing have been improved, we think that batch computing also needs a variety of improvements. No matter in the task scheduling layer or in the data Shuffle layer, a lot of work needs to be improved in terms of fault tolerance and ease of use.

For the reason of the space, here are two main points to share with you:

Unified API Stack

Unified SQL scheme

Let's take a look at the current situation of Flink API Stack, which should be known to developers who have studied Flink or used Flink. Flink has two basic sets of API, one is DataStream and the other is DataSet. DataStream API is provided for streaming users, DataSet API is provided for batch users, but the execution paths of these two sets of API are completely different, and even need to generate different Task to execute. So this is in conflict with the unified API, and this is not perfect, not the final solution. First of all, on top of Runtime, there should be a basic API layer for the unified integration of batches and streams, and we hope to unify the API layer.

Therefore, we will use a DAG (finite acyclic graph) API as a unified API layer for batch streams in the new architecture. For this finite acyclic graph, batch computation and flow computation do not need to be expressed clearly. You only need to let developers define different attributes on different nodes and different edges to plan whether the data is a flow attribute or a batch attribute. The whole topology can integrate the unified semantic expression of batch flow, and the whole calculation does not need to distinguish between flow computing and batch computing, but only needs to express its own requirements. With this set of API, Flink's API Stack will be unified.

In addition to the unified basic API layer and unified API Stack, the solution of SQL is also unified in the upper layer. Stream and batch SQL, we can think that stream computing has data sources, batch computing also has data sources, we can simulate both sources into data tables. It can be considered that the data source of stream data is a constantly updated data table, for batch data sources can be regarded as a relatively static table, there is no updated data table. The whole data processing can be regarded as a Query of SQL, and the final result can also be simulated as a result table.

For flow computing, its result table is a constantly updated result table. For batch processing, its result table is equivalent to the result table completed in one update. In terms of SQL semantics, streams and batches can be unified. In addition, both streaming SQL and batch SQL can use the same Query to express reuse. In this way, the streaming batch can be optimized or parsed using the same Query. Even many stream and batch operators are reusable.

The future direction of Flink

First of all, Alibaba should still be based on the nature of Flink to do an all-round unified big data computing engine. Put it on the ground in ecology and scene. At present, Flink is already a mainstream stream computing engine, and many Internet companies have reached a consensus: Flink is the future of big data and the best stream computing engine. The next important step is to get Flink to make a breakthrough in batch computing. Landing in more scenarios, becoming a mainstream batch computing engine. Then further seamlessly switch between streams and batches, and the boundaries between streams and batches become more and more blurred. With Flink, there can be both stream and batch calculations in a calculation.

The second direction is that Flink is ecologically supported by more languages, not only Java,Scala, but even Python,Go, which is used in machine learning. In the future, we hope to use more rich languages to develop Flink computing tasks, to describe computing logic, and to interface with more ecology.

Finally, I have to say AI, because many of big data's computing requirements and data volume are now supporting very popular AI scenarios, so on the basis of the perfect ecology of Flink streaming batches, we will continue to go up and improve the Machine Learning algorithm library of the upper Flink. At the same time, Flink will also learn from mature machines and integrate them with in-depth learning. For example, you can do Tensorflow On Flink to integrate big data's ETL data processing with machine learning Feature computing and feature computing, and training computing, so that developers can enjoy the benefits of multiple ecology at the same time.

For more information, please visit the Apache Flink Chinese Community website.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.