Shulou(Shulou.com)06/03 Report--

This year, real-time streaming computing technology began to enter the mainstream, major manufacturers are sparing no effort to try the new streaming computing framework, real-time streaming computing engine and API such as Spark Streaming, Kafka Streaming, Beam and Flink continue to be popular. Alibaba has improved Flink since 2015 and created an internal branch, Blink, which currently serves a large number of core real-time businesses such as search, recommendation, advertising and ants within Alibaba Group. On December 20, the Flink Forward China Summit hosted by Alibaba was held in Beijing National Convention Center. Technical experts from Ali, Huawei, Tencent, Meituan Dianping, Didi, Byte Jump and other companies shared their Flink-based application and practical experience with participants. In the keynote speech at the conference, Zhou Jingren, vice president of Alibaba Group, announced that Alibaba's internal Flink version Blink will be officially open source in January 2019! Ali hopes to further deepen its linkage with the Flink community through Blink open source, and promote more domestic small and medium-sized enterprises to use Flink.

At the meeting, I conducted an exclusive interview with Jiang Xiaowei, a researcher of Alibaba Computing platform Division, who shared his views on the next generation real-time streaming computing engine and answered questions such as the important new features of Blink, the relationship between Blink and Flink after open source, and the follow-up planning of Blink.

Alibaba and Flink

With the advent of the era of artificial intelligence and the outbreak of the amount of data, in the typical big data business scenario, the most common approach of data business is to select batch processing technology to deal with all data, and use streaming computing to process real-time incremental data. In many business scenarios, the user's business logic is often the same in batch and flow processing. However, the two sets of computing engines that users use for batch and stream processing are different.

As a result, users usually need to write two sets of code. There is no doubt that this brings some additional burden and cost. Alibaba's commodity data processing often needs to face two different business process problems of increment and full volume, so Alibaba is thinking: can there be a unified set of big data engine technology? users only need to develop a set of code according to their own business logic. In this way, in a variety of different scenarios, whether it is full data or incremental data, or real-time processing, a set of solutions can all be supported. This is the background and original intention of Alibaba's choice of Flink.

At that time, the scale and stability of Flink had not yet experienced practice, and the maturity was open to question. Alibaba's real-time computing team decided to set up a Flink branch Blink within Ali, and made a lot of modifications and improvements to Flink to adapt it to the super-large-scale business scenario like Alibaba. To put it simply, Blink is the Alibaba internal version based on open source Flink developed by Alibaba.

Alibaba's Flink-based platform was officially launched in 2016, starting with Alibaba's search and recommendation scenarios. At present, all the businesses of Alibaba, including all subsidiaries of Alibaba, have adopted a real-time computing platform based on Flink.

At present, this Flink-based real-time computing platform not only serves within Alibaba Group, but also provides Flink-based cloud product support to developers through Aliyun's cloud product API.

The following content is sorted out from the interview with Jiang Xiaowei at the front of AI.

The opportunity for open source

Me: why did you choose to open source Blink now? What are the considerations? What kind of time is the most appropriate time to open source?

Jiang Xiaowei: in my opinion, there are several factors: the first factor is that we have been trying to push Ali's improvements on Flink back to the community in the past few years, but the community has its own pace, and very often we may not be able to push our changes back in time. For the community, there needs to be a consensus to better ensure the quality of open source projects, but at the same time it will lead to a slower push. After years of accumulation, the gap between our side and the community has become larger. Blink has some nice new features, such as batch processing, that are not available in the community version. Over the past period of time, we have been hearing calls from people asking when and if Blink can open source. We have two ways, one is to push it back slowly and then give it to the user. But we don't think it's best for the community to wait like this. We still hope to get our code out as soon as possible so that everyone can use it as much as possible. So in the last six months, we have been preparing to organize the code for open source.

There are several advantages in choosing to open source at this point in time: the first benefit is that the code we open source has been tested by huge traffic within Ali, such as double 10 and double 12, so that we have greater confidence in its quality. this is a very big benefit. The second benefit is that the Flink Forward conference is held for the first time in China. Open source on such an occasion shows Ali's firm support for the Flink community. This is a good occasion. Mainly based on these considerations.

Blink or Flink? It won't be a problem.

Me: will the open source version of Blink be consistent with the Blink used internally by Alibaba?

Jiang Xiaowei: what is about to open source is the online version of Alibaba double Twelve, and there will be some small improvements.

Me: what will be the relationship between two open source projects after Blink is open source? Will Flink and Blink be maintained by different teams in the future?

Jiang Xiaowei: open source means that we are willing to contribute the Blink code, but these two projects are one project. One thing to clarify is that we will release all the code for Blink so that everyone can see it, but at the same time, we will work with the community to discuss how it is most appropriate for Blink to enter Flink. Because Flink is a community project, we need the consent of the community to enter the Flink as a branch, or as a change Merge to the project. I would like to emphasize that as members of the community, we need to discuss with the community before we can decide on this matter.

Blink will never become another project, if the subsequent entry into Apache must be part of Flink, we have no interest in setting up another flag, we will always be a part of Flink and will firmly support Flink. We are very willing to contribute the Blink code to everyone, so we will release the Blink code in January next year, but during this time we will also discuss with the community what form to enter Flink is the most appropriate and how to contribute is the most desirable way for the community.

We hope that after Blink is open source, we will work with the community to gradually push the good parts of Blink back to Flink and become part of Flink. We hope that eventually Flink and Blink will become one thing, and Alibaba and the whole community will work together to maintain them. Instead of dividing it into two things, giving users the difficulty of choice, this is not what we want.

Therefore, in the future, users will not face the problem of whether to migrate Flink to Blink after Flink has been deployed, and enterprises do not need to choose between Flink and Blink. Blink and Flink will be the same project. Blink open source has only one purpose, and that is to hope that Flink can do better.

What has Blink improved?

Me: can you focus on the more important new technical features of the upcoming open source version of Blink? Compared with the latest release of Flink, what optimizations and improvements have been made to Ali's Blink?

Jiang Xiaowei: Alibaba real-time computing team has not only made many improvements and optimizations to Flink in terms of performance and stability, but also made a lot of innovations and improvements in core architecture and functions. Over the past two years, many updates have been pushed back to the community, including Flink's new distributed architecture.

At present, there are several differences between our Blink version and the community version. The first is stability. We have made some optimizations. In some scenarios, it will be more stable than the community version, especially in large-scale scenarios. Another big difference is our new Flink SQL technology stack, which is much more powerful than the community version in terms of functionality, especially batch processing. It supports almost all the syntax and semantics of the current standard SQL. In addition, in terms of performance, our version has a great advantage in performance, whether in streaming SQL or batch SQL. Especially in terms of the performance of batch SQL, the performance of the current Blink version is more than 10 times that of the community version, and compared with Spark, the performance of Blink in scenarios such as TPCDS can also reach more than 3 times. If users have a strong demand for batch processing or SQL, users will get a lot of benefits from this version.

The Application of Blink in Ali

Me: please introduce the use of Blink in Ali. What role does Blink play in Ali's big data architecture at present? What business and application scenarios are mainly used within Ali?

Jiang Xiaowei: now on Alibaba's big data platform, all real-time computing is already using Blink;. At the same time, in addition to real-time computing, Blink is also used for batch processing in some streaming and batch integration scenarios. We also have an exploration in the machine learning scene, called Alink, this project is an improvement to Flink Machine Learning Library, in which a large number of algorithms are implemented, all of which are based on Flink for real-time machine learning. Alink has been proved to have great advantages in scale in many scenarios. At the same time, we also have some exploration in the graph computing scenario.

Me: how many departments in Ali are using Blink at present?

Jiang Xiaowei: we just made statistics some time ago that about 70% of Ali's technical departments are using Blink. Blink has been growing up in the feedback of users. Blink has made targeted improvements to the problems of data tilt, resource utilization and ease of use reported by internal users.

At present, the most commonly used scenarios in Blink are real-time computing. Alibaba has some businesses that are relatively new and have not yet entered the field of real-time computing. Blink will also be used when these services enter the field of real-time computing.

In terms of batch processing, Ali also has a self-developed batch engine called MaxCompute,MaxCompute that also embraces the Flink ecology and does syntax and semantics compatible with Flink. In the future, the entire Ali computing system and platform will be integrated into the same ecology.

Subsequent planning

Me: what other plans does Ali have for Blink next? It includes technical improvement, landing application, update and maintenance, community and so on.

Jiang Xiaowei: technically speaking, today we have announced the achievements of Flink in batch processing. Next, we will continue to invest in technology, and we hope to see a big bright spot in technology every few months. The next bright spot should be the machine learning scene. There is a series of work to be done to support machine learning well, including the function, performance and ease of use of the engine. We are already in the process of internal discussion and progress, and you should see some results in the next few months. We are also discussing something with the community. In addition to machine learning, we also have some explorations in graph computing, including better support for incremental iterations. After doing all this, it can be considered that Flink has been relatively complete as big data's computing engine.

At the same time, we also focus on the ecology of Flink, including the relationship between Flink and other systems, ease of use and so on. For Flink to be really good, it needs not only its own powerful function, but also the whole ecology to be very strong. In this section, we will even work with some ISV to see if we can provide a better solution on top of Flink and further lower the threshold for users.

On the community side, we hope to fully integrate Blink into the Flink community and work together to operate the Flink community, so that Flink can be really used on a large scale in China and even around the world.

In terms of application, real-time streaming computing actually has many potential application scenarios, but some of them may not be very familiar with, we will do some promotion to these scenarios. Take real-time machine learning as an example, it can often bring us a greater effect than ordinary machine learning. Last year, real-time reinforcement learning improved our search by more than 20%. In addition, in the field of security (such as real-time Fraud Detection), monitoring and alarm, and in the field of IoT, real-time stream computing has a very wide range of applications. These Flink may have been done by now, but you don't realize that Flink can bring you such commercial benefits.

Me: how will the changes and updates made by Ali on this basis be pushed back to the community version after Blink is open source?

Jiang Xiaowei: our ideal way is that the internal version of Ali is the Flink version of the community plus some customized plug-ins. There is no need to modify Flink itself, but to add to Flink. For example, if the part that interacts with Ali's internal system is not applicable to the community, it will be kept internally. We hope that these changes will not change the Flink code, but will be added to Flink in the form of plug-ins. In the end, changes that are useful to all companies will be made in the Flink code itself, so that all companies that use Flink can benefit from it, while the part of docking Ali's internal system is only used in Ali.

The debate of the next Generation Real-time streaming Computing engine

Me: first of all, when many people mention real-time streaming computing engines, they will compare Spark with Flink. What do you think of the next generation of real-time streaming computing engines? What is the most important development direction of real-time streaming computing engine in the future?

Jiang Xiaowei: at the beginning, Spark and Flink had the same dream of share. They both hoped to unify streaming and batch processing with the same technology, but they took two completely different paths. The former is based on the technology used for batch processing and tries to support stream computing on top of batch processing. The latter believes that stream computing technology is the most basic, supporting batch processing on the basis of stream computing. Because of this difference in architecture, there will be some subtle differences in what the two can do in the future. For example, in low-latency scenarios, the micro-batch-based approach of Spark requires synchronization will have additional overhead, so it can not achieve the maximum latency. Flink already has a great advantage in the low-latency scenarios that big data deals with. After our exploration, Flink has also made great breakthroughs in batch processing, and these breakthroughs will be fed back to the community. Of course, for users, one more choice is always good, different technologies may bring different advantages, users can choose according to the needs of their own business scenarios.

In the future, in the direction of big data, machine learning is gradually developing from batch processing and offline learning to real-time processing and online learning, and the same thing is happening in the field of graph computing, for example, real-time anti-fraud is usually done by graph computing, and these fraud events occur in real time and continuously, and graph computing is becoming real-time.

However, in addition to big data's field, Flink also has its unique advantages in application and micro-service scenarios. Application and micro-service scenarios require very strict latency, reaching the level of 100 milliseconds or even 10 milliseconds, which can only be achieved by the architecture of Flink. I think applications and micro services are actually very big areas, maybe even bigger than big data. This is a very exciting opportunity. These are all the application fields that we hope to broaden.

Me: in terms of technology, Spark and Flink actually have their own strengths, but in terms of ecology and the companies behind, Flink is weak, so how will Ali help Flink in the future in the area of ecological and corporate support?

Jiang Xiaowei: this time Ali held Flink Forward China is one of the important moves to promote Flink ecology. In addition to Flink Forward China conferences, we also hold various offline Meetup from time to time and put a lot of effort into building a Chinese community, including translating Flink English documents into Chinese, creating Flink Chinese forums, and so on. In the vertical area, we will look for some partners to package Flink in some solutions for users to use.

Me: about the neutrality of open source projects. Ali is now vigorously promoting the application of Flink open source projects and the development of the community, but other companies in the industry (especially those that may compete with Ali in other businesses) may still have some doubts about the neutrality of the community when considering whether to adopt Flink. How does Ali think about this?

Jiang Xiaowei: Ali itself will make great efforts to promote the development and growth of the Flink community, but we also very much hope that more enterprises and people will join the community and work with Ali to promote community development. Ali's hosting of the Flink Forward China Summit is an opportunity for more companies to participate. The Ali family alone is unable to do Flink ecology. I hope you can see what we are doing and then dispel such doubts. We will use our actions to show that we really want to make the Flink community bigger, and we will not be selfish in this matter.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.