Shulou(Shulou.com)06/01 Report--

Flink+HBase scene solution is what, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain for you in detail, people with this need can come to learn, I hope you can gain something.

Real-time computing scenario solution provided by Flink+HBase.

Real-time Computing Market Competition Analysis-- traditional manufacturer

In the real-time computing market, the layout and start of traditional IT manufacturers such as IBM and Oracle are relatively early, so they can provide very rich scenarios, whether 2B or 2C, these traditional manufacturers have very rich products. However, the products they offer are often expensive and require localized deployment. In addition, in the process of cloud, the overall performance of IBM and Oracle is also poor, so these traditional manufacturers are also actively looking for more cloud scenarios.

Real-time Computing Market Competition Analysis-- Cloud manufacturer

In addition to the traditional IT giants mentioned above, many cloud computing vendors are also actively laying out real-time computing. For example, Google has been investing in big data-related fields, and it also entered the field of real-time computing in 2014. Google's main product in the field of real-time computing is Dataflow. Microsoft Azure and AWS entered the field of real-time computing in 2015 and 2016 respectively. Aliyun and Huawei both entered the field of real-time computing in 2017, while Tencent Cloud just started its layout in 2018. In short, cloud manufacturers start relatively late in the field of real-time computing, and the scenarios involved are relatively shallow, but they have advantages in price comparison, so the future development space is huge.

Ecological Analysis of Open Source Technology in the Industry

At present, in the field of real-time computing and stream computing, open source technology ecology has mainly produced three generations of products: the first generation is represented by Storm, the second generation is represented by Spark, and the third generation is represented by Flink. This paper focuses on that Flink,Flink is a stateful real-time computing processing engine, and officially, because Flink has state, it is very suitable for scene-based solutions in event processing, and can be further evolved into a micro-service framework, so Flink is also a very suitable technology for micro-service scenarios.

At the same time, you can also see that Flink is also the most active project in the entire Apache community during the period from last year to the beginning of this year. However, the Aliyun Real-time Computing team also found that Flink is still a technology biased towards the PaaS layer, and customers cannot directly solve their scene requirements by using Flink. In front of Flink, they need tools like Kafka to import data, and behind it, they need some storage tools like HBase to help achieve data storage. Therefore, to solve the problem of real-time computing scenarios through Flink, you still need a whole set of ecological help.

Flink containerization solution-Architecture

The Aliyun real-time computing team has been thinking about how to integrate these open source technologies so that they can really solve the problems faced by their customers and help them solve the "last kilometer" problem. Because PaaS layer tools like Flink can not help customers solve this "last kilometer" problem, then the product will not be accepted by customers. In Aliyun's Flink containerization solution, the underlying layer is built on Google's K8S container. In addition to importing Flink's real-time computing engine, components such as Kafka upstream of Flink and HBase downstream are also added in the architecture. Therefore, the Flink containerization solution can provide all the data processing capabilities under the K8S framework of Google. If the customer has other needs, you can also add the corresponding services in the way of K8S Orchestra.

In the upper layer of the architecture, Aliyun Flink containerization solution hopes to provide SaaS layer services closer to users. In terms of real-time computing + Flink or HBase, Aliyun Real-time Computing team made user portraits of the customers they served, and finally abstracted containerization services for several major industries, such as rule engines for security industries and urban brains, decision engines for banks and other risk control areas, and scenarios for voice processing, natural language processing and video analysis. And there are real-time online data analysis and real-time artificial intelligence scenarios.

In addition, Aliyun Real-time Computing team also hopes to integrate some "PaaS+" services that are closer to customers on the basic service layer, so it also provides customers with system administrator interface and user development interface in the architecture of Flink containerized solution. Therefore, for Aliyun's Flink containerization solution, it is mainly in accordance with the above ideas to design more detailed, mirror technology superimposed solutions and products.

Real-time containerized computing solution-- ecological partner

For Aliyun's real-time container computing solution, we very much hope to introduce more ecological partners, and hope that more partners with rich experience in the vertical field will continue to join us to build more solutions that prefer the SaaS layer on this platform, such as adding more pluggable or easier-to-use solutions. Become the distribution channel or delivery channel of the product or solution, and undertake the channel of expansion, delivery and after-sales service. Aliyun very much hopes to build a real-time containerized computing ecology that integrates Flink, HBase and Kafka products through ecology or channels, and to build an end-to-end closed loop of product form.

Typical scenarios of Flink Real-time Computing

The following figure shows a typical scenario of Flink real-time computing in Aliyun. The product was officially launched in October 2018. Half a year after its launch, through the user portraits and analysis of customers by Aliyun, it is found that many real-time computing products on the cloud are still relatively shallow in use. At present, real-time computing products on the cloud can reach three main areas, namely, data analysis, event-driven and data processing.

For data analysis scenarios, the main product is the real-time data large screen. For example, in Ali Shuang 11, there is also a super data large screen display, which includes real-time BI and other scenarios and solutions. For event-driven scenarios, the main products are those related to real-time monitoring and real-time risk control. Because Flink is stateful, it is inherently capable of event processing. In the actual scenario, it is found that many customers will combine a variety of scenarios, such as rule engine, decision engine, indicator monitoring and tuning. Further, there are data processing scenarios. In fact, Flink+HBase belongs to strong computing scenarios, so it is hoped that more strong computing scenarios can appear, such as urban traffic brains. Nowadays, many cities across the country have deployed a large number of traffic cameras, which generate a large amount of video surveillance data every day, so strong computing scenarios are needed to support them. In addition, in areas such as online education, there are also a lot of videos settling down. For real-time data processing, it is necessary to find scenarios that can generate a large amount of data in production or life. Aliyun also hopes to provide more capabilities and services in such scenarios.

Online Education-- Real-time Video Analysis scenario

The Aliyun Real-time Computing team has precipitated many scenarios through in-depth communication with customers. The real-time video analysis scenario of online education is shown in the figure below. Ali Yun has fully explored real-time video analysis in urban brain and other fields, but in civilian-oriented fields, especially in aspects related to daily life, the application of real-time video analysis solutions is not deep enough. As a result, during the period from the end of 2018 to early 2019, the Aliyun real-time computing team conducted several rounds of in-depth communication with China's top unicorn companies in the field of online education. fully tap some of the needs of these customers in the field of real-time video analysis. Generally speaking, it can be considered that there is a strong demand in the field of online education for judging curriculum quality and monitoring curriculum progress through video analysis, while Aliyun believes that real-time capacitive computing solutions such as Flink+HBase+Kafka can play a good role in civil video analysis such as online education.

Online Education-Real-time Forecast scenario

For the field of online education, in addition to the real-time video analysis scenarios mentioned above, there are also real-time prediction scenarios. For example, on the online education platform, there may be a large number of courses open at every time of the day, so a lot of video data will be generated, and there will often be about 5-minute recess between these courses. During the five-minute recess, the online education platform hopes to evaluate the operation of all the network, server and other infrastructure and platform systems, and predict how many courses the platform will be able to open in the next 30 minutes with the help of machine learning, which is also an application scenario of Flink+HBase in Online machine learning. For such a scenario, Aliyun Real-time Computing team is also discussing with customers, hoping to achieve more excellent cases with reference value in this scenario.

Urban brain-- Real-time Video Analysis scene

The picture below shows Alibaba's more urban brain projects. At present, the urban brain has landed in Hangzhou, Shanghai, Haikou and other cities, and its bottom layer is through the combination of real-time computing and HBase, so as to realize the processing and analysis of the whole video stream. These processed video data often come from the high-definition cameras in the city, which can capture a large amount of data in real time and transmit it to the city brain. In order to realize the real-time allocation of traffic lights through the urban brain, we need a real-time data processing platform composed of Flink. After data processing, the eigenvalues and related indicators generated by the whole dynamic video data are stored in HBase, and various types of algorithms are added to analyze the data, so as to realize the intelligent management of the whole urban traffic. This is a typical scene of real-time video analysis and processing, and it is also a practical scene in China's first-tier cities and many provincial capital cities.

Real-time fraud detection (risk control) scenario

The following figure shows the scenario of real-time fraud detection, that is, the risk control scenario. For risk control, there are not only financial risk control, but also marketing risk control and so on. Take the marketing risk control as an example, first, the user's behavior is reported by APP or Web log, and then sent to a message queue, and then calculated in real time through the risk model and rule engine, thus generating some message early warning.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.