Shulou(Shulou.com)05/31 Report--

How to parse the highly available big data service MaxCompute based on Serverless, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

What is MaxCompute

Big Data in Alibaba

First of all, I would like to introduce some relevant background of Alibaba and big data's technology. As shown in the following picture, Alibaba really began to lay out big data's technology a long time ago, and it can even be said that Aliyun was established to help Alibaba solve big data-related technical problems. Today, almost all of Alibaba's BU uses big data technology. Within Alibaba, big data technology is not only widely used, but also very in-depth. In addition, the big data system of the whole group will eventually come together.

Cdn.com/79d321f070f44bdc12462c5862973ac1ae544738.png ">

Overview of Computing Platform

The responsibility of Ali Cloud Computing platform Division is to bring together Alibaba's big data system and be responsible for the storage and computing related R & D work of the entire group. The following figure shows a simple architecture diagram of Alibaba big data platform. The bottom one is Pangu, a unified storage platform, which is responsible for storing big data. Storage is static, and in order to mine the value of data needs to rely on computing power, so Alibaba big data platform also provides a variety of computing resources, such as CPU, GPU, FPGA and ASIC. In order to make better use of these computing resources, they need to be abstracted and managed effectively, and within Aliyun, a unified resource management system is called Fuxi. Based on the resource management and scheduling system, Aliyun has also developed a variety of computing engines, such as general computing engine MaxCompute, stream computing Blink, machine learning PAI and graph computing platform Flash. On top of these computing engines, the platform also provides a variety of development environments, and implements a variety of business based on these development environments.

The following will focus on the general computing engine MaxCompute. MaxCompute is a general distributed big data processing platform. On the one hand, it can store massive data, on the other hand, it can also carry out general calculation based on the data.

First of all, let's look at two numbers. Currently, 99% of Alibaba's internal data storage and 95% of its computing tasks are carried by MaxCompute. In fact, MaxCompute is Alibaba's data center, and the data generated by each BU of the group will eventually be aggregated to MaxCompute, making the group's data assets continue to increase. Secondly, look at a few more indicators. The performance of MaxCompute in BigBench testing is 2.5 times higher than that of the average open source system. Within Alibaba Group, the number of machines in the MaxCompute cluster has reached tens of thousands, and the amount of work carried every day has reached the level of one million. At present, the amount of data stored in the MaxCompute cluster has reached the EB level a long time ago, which is in the leading level in the world. In addition, MaxCompute not only provides services within the group, but also is open to external enterprises. At present, more than 50 sets of industry solutions have been provided. Today, the storage and computing capacity of MaxCompute is growing at a very high rate every year, and with the help of Aliyun's ability, MaxCompute can be deployed not only at home, but also to many overseas countries and regions.

MaxCompute system architecture

The system architecture of MaxCompute is similar to that of big data computing engine in the industry. As shown in the following figure, users can access through some clients, and the data can be transferred to MaxCompute through the access layer. In the middle, there is a management layer, which can not only manage a variety of user jobs, but also carry a variety of functions, such as the most basic SQL compilation, optimization and running functions in big data. In addition, MaxCompute provides distributed metadata services, because without metadata management, there is no way to know exactly what the data is stored. At the bottom of the MaxCompute architecture is where the actual storage and computing takes place.

Second, MaxCompute Serverless and the mysteries behind it

The concept of Serverless is very popular these days, and from the point of view of MaxCompute developers, the popularity of Serverless is both happy and depressing. The good news is that the MaxCompute technical team began to develop similar functions many years before the concept of Serverless was put forward, and the design concept is fully in line with Serverless, which shows that the layout of MaxCompute in Serverless is early, and the technology is advanced. But the depressing thing is that although the MaxCompute team put forward the concept of Serverless a long time ago and recognized the value of this technical route, they were not able to package such capabilities as soon as possible.

Before the concept of Serverless was put forward, big data practiced basically following the steps in the figure below: first, buy a server and build a cluster. Once you have the hardware, you can install the big data processing software "whole family bucket" on it, then import the data, compile the queries required by the business and do calculations, and finally get the results. Of course, each of the above steps is inseparable from the relevant work of operation and maintenance. If you build your own big data system, then the operation and maintenance work will run through all the time, and for the operation and maintenance students, they also need to be on standby all the time to prevent problems in the system.

However, if you look back at the above steps, the real business value is the fourth step of writing queries and doing actual calculations, but other steps will consume a lot of resources and manpower, which is an additional burden for business-oriented companies. Today, Serverless offers an opportunity to help companies eliminate these additional burdens.

In the case of MaxCompute Serverless, it takes only four steps to achieve the required business value. The first step is to open an account and create a project, which is both a data storage carrier and a computing carrier; the second step is to upload data; the third step is to write a query statement and calculate it; and the fourth step is to obtain the calculation results. No operation and maintenance work is required throughout the process, so the development team can focus on the core areas that can generate business value, which is the advantage of Serverless. In fact, it also represents the changes that new technologies bring to daily work, which enable people to focus more on areas that generate core value, without having to care about additional or auxiliary work.

Different from other open source software or big data's practice of providing a big data software package directly to customers, MaxCompute provides big data services, and the services provided can achieve high reliability of 365days) X 24 (hours), providing not only the computing power of big data, but also the ability of data storage, and enables users to truly realize the use of big data technology out of the box.

There are many challenges to be faced in the process of implementing the Serverless service on big data platform. The editor will focus on the following three issues:

How to continuously iterate and upgrade big data service

The trend of big data's Service: automation and intellectualization

Data security

Challenges and solutions in continuous improvement and release

If you want to achieve a big data system, then the topic of continuous improvement and release is absolutely inseparable. This is because users always put forward a variety of new requirements, and in order to meet these needs, we need to constantly upgrade and improve the big data platform, and at the same time, we also need to face the millions of jobs running on the platform every day. And ensure uninterrupted service 24 hours a day. Then, how to ensure the stability and security of the platform upgrade process and make users unaware of the release of new features, as well as the need to ensure the stability of the new version, the absence of Bug and performance fallback problems, and the ability to quickly stop losses after problems, these are all issues that need to be considered. In addition, the contradiction between testing and data security needs to be considered in the process of continuous improvement and release. All in all, achieving continuous improvement and release on the big data platform is like changing the engine on a flying plane.

Faced with the above problems and challenges, Aliyun MaxCompute team has done a lot of work. At present, in the field of big data, MapReduce is not user-friendly enough has become the consensus of the industry, SQL-like big data engine has become the mainstream. For SQL-like processing, the three most critical steps are compilation, optimization, and execution. Therefore, MaxCompute also grasps the crux of the problem, developing Playback tools for compilation and optimization, and Flighting tools for execution. In addition to the above comparison biased towards the two tools used in the test verification phase, MaxCompute also provides tools such as grayscale online when it is actually online.

Playback tool

The background of Playback is that MaxCompute needs to quickly improve the expression ability and performance optimization level of compilers and optimizers. After making major changes to the compiler and optimizer, how to ensure that there is no problem with the compiler and optimizer has become a challenge. The most original way is to put some existing SQL on the compiler and optimizer to execute, and through the manual analysis of the execution results to find the problems in the improvement of the compiler and optimizer. However, manual analysis may take a hundred years to complete, which is obviously intolerable. Therefore, the MaxCompute team hopes to use the computing power of big data computing platform from me to verify the new compiler optimizer.

Its principle is to make use of the powerful and flexible ability of MaxCompute itself, collect all real user queries and put them into MaxCompute system, run user queries in a large-scale and distributed way, and complete the compilation and optimization of these jobs through a new version of the compiler optimizer. Because MaxCompute adopts the popular mode in the industry, it is easy to insert the job plug-in into the task to verify the compilation and optimization task, and it can also be easily exposed if there are problems.

To sum up, on the one hand, Playback uses the flexible and powerful computing power of MaxCompute to analyze a large number of user tasks, on the other hand, it also uses the support for UDF and a good isolation scheme for compilation, optimization and verification. This makes it very easy to verify the new version of the compiler optimizer, and now with the help of the powerful capabilities of the big data platform, tasks that previously wanted to be implemented but could not be done.

Flighting tool

Flighting is a tool for the runtime. In fact, the conventional way to optimize the test of the improved optimizer is to build a test cluster for testing, which is the most conventional and natural way, but it will also cause a great waste of resources, because the cost of the test cluster is very high. Because the task complex load of the test environment is different from that of the real environment, the test cluster can not truly simulate the actual scenario. In addition, in order to validate in a test environment, testers are required to create some data, but the created data may be too simple. Otherwise, you need to take some data from the real scene, but to do so requires not only the consent of the user, but also data desensitization, which is also an extremely troublesome thing. Not only may there be the risk of data leakage, but also there will be some differences between the final data and the real data. In many cases, the problems existing in the new features can not be verified.

MaxCompute's runtime verification tool, Flighting, uses real clusters and environments online. With the help of the ability of resource isolation, 99% of the computing resources in the MaxCompute cluster are handed over to users to run jobs, while the remaining 1% resources are handed over to Flighting tools to verify the new functions of the executors. This approach is very efficient, not only does not need to drag data, but also can be run directly in a production environment, which can easily expose the problems of real actuators.

Grayscale online

The MaxCompute platform has the ability to go online in grayscale, which can rank tasks according to importance, publish them with fine granularity, and support instantaneous rollback, thus minimizing risk.

Iterative process of continuous development and verification of MaxCompute platform

The following figure shows the iterative process of continuous development and verification of the MaxCompute platform, which combines the Playback tool for the compiler optimizer, the Flighting tool for the runtime, and the ability to go online in grayscale. In such a process, the second cycle does not have to wait for the completion of the first cycle before starting, which makes the whole R & D process more efficient. At present, the continuous development and delivery process of MaxCompute has strong technical advantages in compilation, optimization, execution and grayscale, as well as strong technical competitiveness in the whole industry, which enables MaxCompute to better evolve and bring customers more and better functions. At the same time, it also ensures that the MaxCompute platform will not affect the normal use of users in the process of upgrading.

Automatic and intelligent

With the continuous development of artificial intelligence and big data technology, it is necessary to achieve not only high availability but also automation for services. For such an idea, we can look at it from two angles, namely, the service itself and the user operation and maintenance. In view of the service itself, first of all, we need to achieve automation, and further achieve artificial intelligence.

Any service will have a variety of problems, there are a variety of Bug, when there are problems, the traditional solution needs to rely on manpower to solve. The ultimate goal of service automation and intelligence is to find and solve problems completely through the self-healing ability of the system without manual repair when there are problems in the system. The goal of service automation and intelligence needs to be achieved step by step. From the point of view of the user's operation and maintenance, it is transferred from the operation and maintenance work that the user was originally responsible for to the cloud big data platform that completed big data's task, such as Aliyun MaxCompute. At this time, the customer's operation and maintenance work is transferred to the MaxCompute team. The third step is that the MaxCompute team also hopes to extricate itself from the tedious operation and maintenance tasks of the big data platform, realize the automation and intelligent operation and maintenance of the service, and further release the energy of the MaxCompute team and devote all its energy to more valuable work. The purpose of realizing big data service automation and intelligence is to improve service reliability, liberate productivity, and give back to users by doing more valuable things. The basic idea is to collect and monitor the indicators of the service regularly and take timely measures to deal with the abnormal indicators.

In addition to the automation and intelligence of big data's service, it is also critical for user queries to automatically identify optimizable queries and provide optimization suggestions. The SQL query you write may not be the most efficient, and there may be some more effective ways or better technologies, so it is necessary to automatically identify the optimizable parts of the user query and give corresponding optimization suggestions. For this part, it is also the development direction of the industry, and the MaxCompute team is also carrying out corresponding research work at present.

Third, the secret of the low cost of MaxCompute

Low cost makes MaxCompute more competitive, and behind the low cost, it is MaxCompute that gives play to the technology dividend so that the cost is really reduced. In order to make the cost of big data's task lower, MaxCompute is mainly improved and improved in three aspects: computing, storage and scheduling.

In terms of computing, MaxCompute optimizes its own computing engine to improve performance and reduce the number of resources occupied by jobs, both of which can enable the platform to carry more computing tasks, each task can either reduce computing time or job size, and ultimately reduce the resources occupied by computing tasks, so that more resources can be vacated and more computing tasks can be carried.

In terms of storage, there are also a variety of optimization methods. For example, optimize the compression algorithm, use more advanced column storage, classify the data storage, archive the cold data, and collect the worthless data in time.

In the aspect of scheduling, the optimization methods mainly include intra-cluster scheduling and cross-cluster global scheduling in order to reduce the cost.

For the internal scheduling of the cluster, there are many problems that need to be solved. On the one hand, there are many job instances waiting to be scheduled all the time, and on the other hand, there are many resources waiting to run jobs.

Intra-cluster scheduling

Then how to combine the two is very critical, but also a great test of scheduling ability. At present, the scheduling decision frequency of MaxCompute big data platform has reached 4000cycles. The scheduling decision made here is to put a certain size of jobs into the corresponding resources, so that it can not only meet the needs of the job, but also can not waste resources.

In terms of performance, incremental scheduling is needed to make good use of resources; in addition, flexible quota capability is provided to enable multi-users to cut peaks and fill valleys when running computing tasks. Because when you really use the big data platform, it is impossible for everyone to submit tasks at the same time, and in most cases, tasks are often submitted at the wrong peak, so there will be peaks and valleys for the use of resources. through the flexible quota ability, we can try our best to cut the peak and fill the valley, so that the resources can be fully utilized. For the optimal scheduling of tasks, the balance between delay and throughput needs to be considered. In terms of load balancing, some hot spots need to be avoided. In addition, MaxCompute's scheduling decision system also supports task priority and preemption, which can achieve real-time and fairness.

In addition, for complex task scheduling, service and job scheduling are further divided. The so-called task here is an one-time job, while for the service, the data job will be processed continuously for a period of time after startup. In fact, there is a great difference between service and job scheduling. Service scheduling requires high reliability and cannot be interrupted, because once interrupted, the cost of recovery will be very high. On the other hand, the job is more tolerant of interruption, and even if interrupted, it can be pulled up and continued to run later.

Secondary quota group

For the scheduling in a single cluster, this paper focuses on the concept of secondary quota group, which is relatively rare in the general big data platform. Before introducing the secondary quota group, we first introduce the relevant content about the quota group. To put it simply, after the various resources of the physical cluster are made into resource pools, some resources are integrated into a group, which is called quota group. On the big data platform, jobs run in the quota group, and jobs in the same quota group will share all the resources in that quota group. However, there are many problems in the common first-level quota group strategy. For example, if the first-level quota group has 1000 CPU cores and 100 users are sharing the quota group, at some point a user submits a very large job, thus occupying the 1000 CPU cores of the entire quota group, resulting in other users having to wait for all their jobs.

In order to solve the problems in the above situation, the secondary quota group needs to be used. The second-level quota group is further subdivided on the basis of the first-level quota group, setting the upper limit of resources that each user can use. Each primary quota group can be divided into multiple secondary quota groups in order to make better use of resources. In the secondary quota group, resources can be shared, but even if a user needs to use a lot of resources, it can not exceed the upper limit of the secondary quota group resources, so it can better ensure the user experience.

Different from the general big data system, MaxCompute can not only achieve internal scheduling of a single cluster, but also achieve global scheduling between multiple clusters, which is also the embodiment of the powerful ability of MaxCompute. A single cluster may not be able to expand due to various factors, so it is necessary to achieve horizontal expansion at the cluster level.

For applications with a large amount of business, in order to ensure that the business can still run normally when a single cluster downtime occurs, it is necessary to let the business run on multiple clusters to ensure the high availability of the business. At this time, global cross-cluster scheduling capability is needed. When a job comes in, you need to analyze which clusters the job can be scheduled to, and which clusters are currently available and idle. At this time, the problem of multiple versions of data will also be involved, because the same data stored in different clusters will bring the problem of data inconsistency. In addition, for different versions of the same data, we also need to consider the specific calculation and data replication strategy, and Aliyun MaxCompute also made the above strategy more perfect after a lot of experience.

IV. Data security

For an enterprise, data security is the most concern when using big data computing platform. From Alibaba's experience, in terms of data security, there are three main points that users are most worried about: the first point: put the data on the platform, whether others can see it; the second point: put the data on the platform or service. whether the person providing the service will see the data. The third point: to host the data on the big data platform, if there is a problem with the platform, then what to do with the data.

For these concerns of users, MaxCompute can deal with them perfectly, and users basically do not need to worry about the security of their data. First of all, the MaxCompute platform has a perfect authentication and authorization mechanism, and the data belongs to the user, not the platform. Although the platform helps users to store and calculate, it does not have data ownership. Authentication and authorization are needed for both data access and computing. Neither other users nor the platform can see the data without authorization. In addition, authorization has a variety of dimensions, including table level, column level, and job level. Because MaxCompute is a cloud-based big data platform, it naturally has the characteristics of multi-tenancy, so it is necessary to achieve data and computing isolation between tenants. As far as data is concerned, the MaxCompute platform realizes physical isolation, which fundamentally eliminates the problem of data security. In addition, the platform also provides a strict access control policy, users can not view the data of other users. In addition, MaxCompute also provides E2E full-link storage encryption, which is particularly important for financial institutions; it also provides CRC check to ensure the correctness of data, and this part of the capability has been used in Alibaba for many years and has been very perfect; finally, the MaxCompute platform also provides users with data desensitization and other functions to facilitate users to use.

This is the answer to the question on how to analyze the highly available big data service based on Serverless. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel for more related knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.