Shulou(Shulou.com)05/31 Report--

This article introduces how to understand EMAS Mobile DevOps solution Mobile DevOps. The content is very detailed. Interested friends can use it for reference. I hope it will be helpful to you.

1. Introduction of Mobile DevOps

1. What is Mobile DevOps

1) the well-known DevOps

At the time point of 2020, DevOps is no longer a new concept. I believe we all have some understanding more or less, but when we are asked to accurately describe what DevOps is, it seems difficult to explain it clearly. In fact, there is no consensus definition of DevOps in the industry so far, the reason why it is difficult to be accurately defined is that DevOps is actually a collection of ideas or even a set of ideas, and it is difficult to be materialized. The word "DevOps" itself can be literally understood as the whole life cycle of software from Dev (Development) to Ops (Operations), but what is the exact definition of DevOps? Among the many definitions of DevOps, I think that the definition of Azure DevOps [1] is more accurate and specific:

DevOps is a compound word of Dev and Ops. It combines people, process and technology to provide value to customers continuously.

What does DevOps mean to the team? DevOps enables previously isolated roles (development, IT operations, quality engineering, and security) to coordinate and collaborate to produce better and more reliable products.

By adopting DevOps culture, practices, and tools, teams can better respond to customer needs, increase confidence in the applications they build, and achieve business goals faster.

There are several key messages in this definition to sum up:

The combination of ① person, process and technology

② DevOps enables previously isolated roles to coordinate and collaborate

③ DevOps is an idea that requires both culture and automation tools to support it.

④ aims to produce better and more reliable products faster.

2) from DevOps to Mobile DevOps

Server-side DevOps is often discussed about DevOps. Since DevOps is an excellent concept of software delivery, why not apply DevOps to mobile delivery? This is the mobile DevOps we are going to introduce today.

Because of the difference between mobile and server scenarios, mobile DevOps and server DevOps will be very different. It is mainly reflected in the following aspects:

Mobile application automation construction is more complex

Build environment fragmentation

Android and iOS platforms need to build environments based on different operating systems and build tool chains. Even the same platform build tool chains also have version fragmentation, such as Android SDK and Gradle on which Android construction depends, while Xcode and Ruby versions on which iOS builds need support at the same time.

Mobile terminal construction involves data security issues such as certificate hosting.

IOS builds dependent Mac equipment for non-standard equipment in the computer room.

Mac equipment is not a standard server and cannot be deployed in a standard computer room. It usually requires a self-built Mac room, which is also a challenge for operation and maintenance and stability.

Automatic build is an indispensable capability in DevOps, which requires mobile DevOps to solve the above problems of client-side automatic build and one-click packet through technical means.

Mobile end is severely fragmented, and application delivery compatibility is a huge challenge.

Different from the consistency of the server deployment environment, the fragmentation of the mobile application running environment is very serious, and the compatibility test coverage is much more difficult than that of the server. The fragmentation of mobile end is particularly serious in Android system, which is mainly reflected in the following aspects:

Fragmentation of mobile phone model

There are many mobile phone manufacturers and a large number of models in the Android market, and different manufacturers will "optimize" the system at the bottom. In theory, any model test that cannot be covered may face compatibility problems. The following picture shows the latest Android Top model distribution of Baidu Statistical flow Research Institute in October. The market share of Top 10 is less than 15%, which shows the serious fragmentation of models.

Fragmentation of operating system version

The difference of the operating system has a more direct impact on the operation of the application, and it is not uncommon that the upgrade of a large version of the system leads to application incompatibility. every release of a large version of the operating system is a test of application compatibility; while considering compatibility with the new system, you can't give up the users of the old system.

The following figure shows the latest Android version distribution data of Baidu Traffic Research Institute in October. We can see that Android 10.0, which has been released for more than a year, has a market share of less than 50%, and the operating system of 2 years ago is still in the mainstream.

Because of the fragmentation of the end equipment, it is necessary for the mobile DevOps to have the mobile testing ability and automate a large number of real machine compatibility tests.

Long cycle for mobile application release and update

The update rate of the new version may not exceed 50% in 2 weeks, unlike the server that can complete the software release of all servers in a very short time. A long release cycle means it is more expensive to make mistakes, and it may take a long time for a version with Bug to be digested through updates and upgrades.

This requires that Mobile DevOps, on the one hand, has a perfect grayscale publishing mechanism to avoid publishing problematic applications to the user side at one time; on the other hand, once a version of Bug has been issued, Mobile DevOps needs to have hot repair capability, and Bug can be repaired lighter and faster through the release of incremental patch packs.

Mobile applications run on massive mobile devices

Unlike server-side services running in a specific cluster, unified control and operation and maintenance, the running environment of mobile applications is on users' mobile phones, and for super App such as hand-Amoy, it is a massive device of 100 million.

This requires mobile monitoring products to achieve mobile operation and maintenance monitoring through big data technology, and even need remote log function to pull error logs on designated devices to locate and troubleshoot errors.

Based on the above points and with reference to DevOps's definition of software delivery life cycle, the mobile DevOps application life cycle and capability requirements of each stage are summarized as follows:

two。 What is Mobile DevOps?

1) Mobile DevOps is the concrete realization of EMAS Mobile DevOps concept.

First of all, let's introduce EMAS (Enterprise Mobile Application Studio). EMAS is the leading native cloud application research and development platform from Aliyun (mobile App, H5 application, Mini Program, Web application, etc.). Based on a wide range of cloud native technologies (Backend as a Service, Serverless, DevOps, low code, etc.), EMAS is committed to providing one-stop application R & D management services for enterprises and developers, covering the whole lifecycle of development, testing, operation and maintenance, and other applications. For more information about EMAS, please see the EMAS details page of Aliyun's official website.

Mobile DevOps is the concrete product output of EMAS mobile DevOps concept, and it is the axis product of EMAS. It joins all EMAS products to realize the above mobile DevOps concept. Mobile DevOps realizes the linkage and complete closed loop of EMAS products which were originally isolated in each life cycle of application as shown above, and realizes the upgrade of EMAS from mobile middleware platform to mobile R & D platform. Mobile DevOps combines the following EMAS products to form EMAS's mobile DevOps:

R & D domain: Mobile DevOps

Test domain: mobile test

Publishing domain: Mobile DevOps

Operation and maintenance domain: mobile monitoring, mobile hot repair

Operating domain: mobile push, mobile user feedback

2) History of Mobile DevOps

Mobile DevOps is the commercial output version of the group's internal mobile R & D platform. The first proprietary cloud output version was first developed by Aliyun and Amoy team in 2017, and the first public cloud version was launched in April 2020.

The following picture is the development history of Mobile DevOps. It can be said that the development history of Mobile DevOps is actually the development history of mobile R & D technology of Ali Group, which is the precipitation of Alibaba's concept of mobile technology and engineering R & D in the past ten years.

3) the present situation of Mobile DevOps

Proprietary cloud has begun to take shape

Mobile DevOps private cloud is mainly aimed at large customers, especially those who are doing digital transformation. These customers have high requirements for security and can only accept the mode of proprietary cloud deployment. At the same time, they are also willing to invest in improving R & D efficiency.

Mobile DevOps is officially exported as a proprietary cloud scene in 2018. At present, it has created value for dozens of major customers in many industries, enabling the digital transformation of enterprise R & D process.

Free public test of public cloud

Compared with proprietary cloud, Mobile DevOps public cloud is more aimed at small and medium-sized enterprises, these customers have demands for improving R & D efficiency, but are sensitive to price, and public cloud is a good way to undertake; at the same time, some outbound businesses within Ali Group (such as exclusive nails) cannot be used for mobile DevOps based on the group's internal R & D platform, and Mobile DevOps public cloud is also a good choice.

Mobile DevOps Public Cloud has been officially free of charge for public testing since 2020.07, and has now served many small and medium-sized and micro customers, as well as exclusive nails, government nails, duck singing and other customers within Ali Group.

Second, cloud native Mobile DevOps

Compared with proprietary clouds, building cloud native Mobile DevOps in public cloud scenarios faces more technical challenges. This chapter will share with you our thoughts, challenges and solutions in the process of building cloud native Mobile DevOps.

1. Why do you need Mobile DevOps for public cloud

1) provide inclusive Mobile DevOps services to small and medium-sized customers.

Although proprietary cloud deployment has the advantages of exclusive and private network security isolation, the high cost of proprietary cloud delivery is destined to be accepted only by high-end players in the industry. The cost input of private cloud Mobile DevOps is evaluated as follows:

One-time investment: one-million purchase cost

Continuous investment: at least 30 W / year server cost + 20 W / year manpower maintenance cost

Based on the above cost calculation, the input costs of proprietary cloud in the first, second and third year are 150W, 50W and 50W respectively, which is unacceptable to small and medium-sized customers.

Aliyun, as the infrastructure of the new era and the hydropower and coal of the new era, it is necessary to provide inclusive cloud services for small and medium-sized enterprises other than more big customers. The public cloud Mobile DevOps is exactly in line with this concept. Based on the advantages of cloud native elastic expansion and postpaid, it can greatly reduce the cost of using Mobile DevOps for small and medium-sized customers. At the same time, the public cloud scenario provides a DevOps R & D process that is more suitable for target customers according to the characteristics of small and medium-sized customers.

2) Linkage EMAS product line to provide one-stop mobile R & D platform for developers

The launch of public cloud Mobile DevOps can effectively link EMAS's existing mobile testing, mobile monitoring, mobile hot repair and other products to enable EMAS to cover the whole life cycle of applications, complete the upgrade of EMAS from mobile middleware to mobile R & D platform, and enhance user experience and stickiness.

Compared with the traditional self-built CI/CD platforms such as Jekins and Gitlab Runner, EMAS one-stop mobile R & D platform has obvious advantages in cost, high availability and technical support, and can cover application construction, testing, release, operation and maintenance, and operation lifecycle management, compared with the traditional self-built CI/CD "chimney" independent open source systems, and has obvious advantages in R & D collaboration efficiency.

two。 Challenges facing Public Cloud Mobile DevOps

Compared with the scenarios where private cloud private network is deployed and used by internal employees, Mobile DevOps in public cloud will face more technical challenges, mainly reflected in the following aspects:

1) Security

Tenant isolation

The first problem facing public clouds is tenant isolation, where different customers not only use shared resources at the same time, but also cannot see each other's data. For building this scenario, in addition to the construction tasks of different customers may affect each other, the construction environment also involves the user's code, certificates and other private information, so there must be a perfect scheme to ensure the isolation of the user construction environment.

Private data security such as codes, certificates, keys, etc.

If there is a construction, it must involve user code, certificate, key, these data are extremely private data, public cloud storage, transmission, use of any problems may lead to heavy losses to users.

External attack

Because the public cloud is exposed to the public network, anyone can use it, and there is also a risk of malicious hacker attacks. In particular, the construction scenario involves a large number of custom execution commands. There must be a sound mechanism to prevent hackers from executing malicious custom commands, leaving a backdoor in the construction environment.

2) High availability

Flexible expansion and reduction must be supported

When the scale of public cloud business grows, it needs to be able to expand and scale up rapidly to adapt to business growth, otherwise it will lead to abnormal services. This requires cloud products to conform to the distributed architecture in terms of technical implementation, especially to support stateless rapid expansion of the cluster.

Build the stability of the environment

The build environment should be stable and avoid the destruction of the build environment caused by attacks or abnormal uses, such as environment variables, build tool chains, etc.

High-standard SLA, real-time online, never downtime

High standard SLA is not only a commitment to customers, but also an awe of Aliyun brand.

3) scalability

The diversity of application architecture leads to great differences in the construction process.

The number of proprietary cloud customers is limited, and there are perfect technical support services for KA customers, so the differences in applications are limited and there are special personnel to support access. However, there are many customers in the public cloud environment, and the diversity of application architecture puts forward higher requirements for the versatility and expansibility of the system.

Diversification of R & D process

There are differences in R & D team size, R & D culture and R & D process among different customers in public cloud, which also puts forward higher requirements for the scalability of Mobile DevOps R & D process.

3. Our solution

In view of the above challenges faced by public cloud Mobile DevOps, we use technical means to solve them from the following two aspects:

1) General architecture based on pipeline

The pipeline architecture will make the construction universal, customize the construction process based on the pipeline, and expand the pipeline business capability based on the task plug-in, which solves the above scalability problem very well. This architecture has the following features:

General construction architecture to support the building capability of the whole platform

Custom orchestration construction process based on YAML

Visual layout of pipeline

Pipelining supports unlimited extension of task plug-ins

2) Building clusters based on containerization / virtualization

Using Linux / Mac Os solution can thoroughly solve all kinds of security and stability problems caused by resource sharing. Each build task starts a new container / virtual machine to run, and the container / virtual machine is destroyed immediately after the construction task is completed. Not only can the running environment between tasks be effectively isolated, but the construction environment is also "often new", which can effectively avoid the destruction of the construction environment. In addition, building a stable stateless containerization / virtualization construction cluster can ensure the high availability of the construction service.

In the following third and fourth chapters, we will elaborate on these two points and decrypt their design architecture and technical details.

III. General construction architecture based on pipeline

1. Technical pre-research

In fact, there are many competitor products based on pipeline design in the industry, especially many similar products abroad, such as Azure DevOps Pipeline and Github Actions, which have many advantages over other products in terms of feature richness, ease of use, documentation and user scale.

Azure DevOps, formerly known as Visual Studio Team Services (VSTS), is a software research and development collaboration platform with a history of more than ten years. Its Azure Pipeline product was released in April 2018. Github Actions product, released in August 2019, is a heavyweight product released by Microsoft after its acquisition of Github. Generally speaking, both are relatively new platforms, and Azure Pipeline has been around for no more than 2 years.

An interesting phenomenon was found in the pre-research. Because Github is already a subsidiary of Microsoft, the design concepts of the two assembly line products are not only similar, but also found that their Mac virtualization solutions are shared with each other, even the Mac virtualization cluster computer room is also shared. Github Actions is more concise and elegant than Azure Pipeline. In addition, Github Actions still continues the open source style of Github, and its pipelined plug-ins are open source. Although it has been online for only more than a year, there are 5000 + open source plug-ins. This is a gold mine from a plug-in point of view, and if these plug-ins can be used directly in Mobile DevOps, the basic pipelined functional plug-ins will be aligned with the open source community. Considering the possibility of supporting these open source plug-ins in the future, the Mobile DevOps design architecture also embraces the Github Actions of the open source community.

two。 The core concept of assembly line

Trigger

Trigger, actively trigger a pipeline execution

Pipeline

Pipeline, the smallest unit that is triggered to run. A pipeline can contain one or more Job

Job

Job is the smallest unit scheduled. According to the execution environment to which Job is scheduled, it can be divided into Agent (build cluster) and Agentless (server) Job.

Multiple Job can be run in parallel without dependency, or can be executed sequentially. The previous relationship of multiple Job can be represented by a DAG diagram.

Each Job can contain one or more Step

Step

Step is the smallest unit to be executed. Each Job consists of multiple sequentially executed Step

Task

Task is a task plug-in with predefined specifications and functions, which can be declared and referenced in Step. A Step contains only one Task.

3. Technical architecture of assembly line

The pipeline consists of the following core systems:

1) Pipeline process engine

Responsible for pipeline trigger, scheduling, state transfer execution, as well as pipeline metadata information maintenance.

Pipeline trigger module

Trigger module is responsible for triggering the execution of a pipeline, supporting manual, timer, event (git event,webhook callback, etc.) three trigger modes. Trigger is the only entrance to pipeline execution. At this layer, the checksum check of the caller can be done, and different trigger parameters can be passed in to control the pipeline execution and scheduling process.

Pipeline choreography module

Pipeline choreography defines a set of DSL language which is used to describe a pipeline. Based on this set of DSL language, a pipeline that can be scheduled and executed can be accurately defined.

Pipeline execution module

The pipeline execution module mainly ensures that all Job in the pipeline are executed in parallel or sequentially according to the correct dependency relationship, and updates the real-time status of the pipeline flow in real time.

2) Job scheduling engine

Job is the smallest unit scheduled in the pipeline, and the Job scheduling engine is mainly responsible for scheduling every Job generated from the pipelined process engine to the correct cluster machine.

3) Integration engine

There are two types of task plug-ins in the pipeline, one is Agent tasks, such as Android and iOS builds, which require a specific build environment, so it is natural to think that they will be scheduled to the build machine by the Job scheduling engine There are also Agentless tasks, such as approval, notification, external system calls, etc., which can be completed on the ordinary server side and will be scheduled by the Job scheduling engine to be executed on the integration engine without consuming valuable construction resources. Most Agentless tasks are related to external service integration.

4) Channel channel service

The Channel channel is mainly responsible for building the communication link and protocol implementation between the cluster and the server. The main functions are as follows:

Build a cluster to request unified authentication

For security reasons, the construction cluster is in a different VPC from other micro-services. It is completely isolated through the network to ensure that the construction cluster cannot directly access the server private network. Based on this background, the construction cluster access server in the above "pipeline technology architecture diagram" takes the public network HTTPS request, which requires authentication to the builder request, and the Channel channel is the closing of the authentication server.

Build a cluster to request a unified closing

To build a cluster, you need to keep the heartbeat, report status, pull tasks and report the execution status of tasks in real time with the server. Channel is the end of these requests and is responsible for assigning requests from different businesses to different micro-services.

5) build a cluster

The construction cluster is mainly responsible for pulling and executing Agent class construction tasks, and the services running in the construction cluster are responsible for starting an isolated build environment that matches the task type:

Launch Docker Container under Linux platform

Android is built on the Linux platform. The Docker containerization solution under the Linux platform is the best choice for environment isolation. Launch the serverless Docker container based on ACK serverless (Aliyun public cloud K8S product), and automatically destroy and recycle it. Cloud-based native ACK serverless maximizes the flexibility of building a cluster, consumes almost no computing resources, and greatly controls the construction cost.

Start the virtual machine under the Mac OS platform

Due to the ecological limitations of Apple, the construction of iOS and Mac App can only be carried out under the Mac OS system, while there is no mature Docker container solution for Mac OS to use. Finally, we achieve environment isolation based on the virtualization solution. We have built a cloud-based Mac virtualized cluster, which completely pooled Mac physical resources, and can quickly complete the elastic expansion and scaling of the cluster, which is completely in line with the concept of cloud nativeness. Each build dynamically creates a virtual machine from the virtualized cluster and destroys it immediately after building.

It is worth mentioning that Mac virtualization cluster is our technical advantage, the following chapter 5 we will detail the practice of Mobile DevOps in the direction of Mac virtualization cluster.

4. Mac Virtualization Construction Cluster at present, Mobile DevOps's Mac virtualization cluster construction scheme is absolutely in the leading position in China. We "may" be the first domestic DevOps platform to build iOS based on Mac virtual mechanization technology, and there are few domestic manufacturers to support iOS construction. The essential reason is actually the limitation of Mac virtualization technology: traditional Mac physical bare metal construction can only be used in internal environment. There are no conditions for open public cloud services at all. Mac virtualization to build clusters is the technical advantage of Mobile DevOps.

1. Virtualization scheme selection

Due to the limitation of the kernel of the Mac OS platform, the current containerization scheme of the Mac OS platform is extremely immature, and the environment isolation of the Mac OS platform can only be taken by virtualization.

Choice of Virtualization Type

The two types of virtualization solutions are shown in the figure below, both of which are implemented based on Hypervisor. The comparison between the two schemes is as follows:

Virtualization scenario 1:

Non-hosting OS is directly based on Hypervisor virtualization VM, which has high resource utilization and is more suitable for cloud service virtualization solution.

Higher requirements for hardware compatibility

Virtualization scenario 2:

Virtualizing the VM based on Hypervisor on the OS of the host is more suitable for desktop users.

Due to the host OS, the hardware compatibility is better

Based on the consideration of our Mobile DevOps to provide public cloud services, option 1 can more effectively improve resource utilization, and hardware compatibility can be avoided as long as we choose appropriate hardware products.

Ecological Security Compliance of Apple

Apple is ecologically closed and has many security compliance restrictions. The Mac platform has the following legal compliance restrictions:

1.MacOS must run on Apple hardware

two。 For commercial purposes, only one macOS instance is allowed to run on an Apple hardware

From the comparison of the above four virtualization schemes, only scenario 4 has both Apple ecological compliance and compatibility, and option 4 is actually the virtualization solution 1 we chose in the previous section. Based on the above type of virtualization and Apple ecological security compliance and compatibility considerations, we finally choose the above solution 4.

two。 Virtualized Cluster of Cloud Architectur

To provide public building services on the cloud, it is not enough to have a virtualization solution alone. A virtualization cluster solution that meets the cloud architecture is also needed to meet Mobile DevOps's demand for building clusters:

Pooling of ① Mac hardware resources-each Mac resource in the cluster should be stateless, and all Mac hardware resources should form a resource pool that can be uniformly allocated and scheduled by the cluster.

Elastic expansion and reduction of ②-the scale of public cloud business is flexible, which requires that virtualized clusters can also adapt to business scenarios and can quickly and flexibly scale up to keep up with the growth of business.

③ High availability-in the event of damage to individual Mac hardware devices, the cluster can quickly and automatically respond to assign tasks to new virtual machines, improving the success rate of task execution.

From single virtual machine to virtual machine cluster, in addition to the above Mac hardware resource pooling, we also need to solve the newly introduced distributed storage and distributed network problems after hardware resource clustering, from virtualized stand-alone to virtualized cluster, as shown below:

V. Prospect for the future

Prospects for the future

At present, the public cloud Mobile DevOps is still under public trial, and there are still many directions that need to be worked on:

Increase the ability to build intelligent error analysis and prompts. In the case of a large number of public cloud users, it is a huge labor cost to build error answers. Later, technical means such as keyword matching, big data analysis, and even AI automatic error classification are needed to directly indicate the causes of construction errors and reduce the cost of manual answer.

Strengthen more interaction with other EMAS products, so that Mobile DevOps series complete application research and development life cycle

Maintain better affinity with the community. Support Github Actions, Azure Pipeline and other platforms for pipelined migration to Mobile DevOps; task plug-ins directly support Github Actions 5000 + open source plug-ins, enjoy open source community dividends

Strengthen the ability to be integrated, so that the Mobile DevOps mobile R & D platform can be better integrated into the customer's existing R & D process

Deeply optimize the efficiency of application compilation and construction, and reduce the time of application construction. The ultimate goal is to make the time of building applications on the cloud much shorter than that of building locally, so that developers can directly feel the advantages of building on the cloud.

About EMAS mobile DevOps solution Mobile DevOps how to understand to share here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.