Shulou(Shulou.com)06/03 Report--

Preface

With the rapid development of computer technology and Internet, VOD technology is favored by education, entertainment and other industries because of its good human-computer interaction and streaming media transmission technology. at present, the product lines of cloud computing platform manufacturers are becoming more and more mature. If you want to build VOD applications and bid farewell to slash-and-burn cultivation, going directly to the cloud will remove various obstacles such as hardware procurement and technology. Take Aliyun as an example:

This is a very typical solution, object storage OSS can support massive video storage, capture and upload video is transcoded to adapt to a variety of terminals, CDN accelerates the speed of terminal devices to play video. In addition, there are some content security censorship requirements, such as porn detection, terrorism detection and so on.

In VOD solutions, video transcoding is the subsystem that consumes the most computing power. Although you can use special transcoding services on the cloud, in many cases, you will choose to build your own transcoding services. For example:

You have deployed a set of video processing services based on FFmpeg on the virtual machine / container platform, can you make it more flexible and more available? Your requirements are only simple transcoding requirements, or some extremely lightweight requirements, such as obtaining the GIF of the first few video frames on OSS and the length of time to obtain video or audio. It is cheaper to build your own. You have more advanced custom processing requirements for audio conversion in various formats or customization of sampling rates and audio noise reduction. For example, after video transcoding is completed, you need to record the transcoding details to the database, or automatically preheat the hot videos to the CDN after transcoding is completed, so as to relieve the pressure on the origin server. You have the need to process a large number of videos concurrently. There may be a variety of operations in the custom video processing process, such as transcoding, watermarking and generating the video home page GIF. Subsequently, new requirements are added to the video processing system, such as adjusting the transcoding parameters. It is hoped that the release and launch of the new features will not affect the online service. Your video source files are stored on NAS or ECS cloud disk, and the self-built service can read the source files directly without having to migrate them to OSS.

If your video processing system has the above requirements, or if you expect to implement a video processing system that is flexible, highly available, low-cost, OPS-free and flexible to support any processing logic, then this article is your best practice.

Serverless Custom Audio and Video processing

Before introducing the specific plan, let's introduce two products:

Function Computing: Ali Cloud function Computing is an event-driven fully managed computing service. Through function calculation, you do not need to manage infrastructure such as servers, you just need to write code and upload. Function calculation will prepare computing resources for you, run your code in a flexible and reliable manner, and provide log query, performance monitoring, alarm and other functions. Functional workflow: functional workflow (Function Flow, hereinafter referred to as FnF) is a fully managed cloud service used to coordinate the execution of multiple distributed tasks. You can orchestrate distributed tasks in sequence, branching, parallelism, etc. FnF will reliably coordinate task execution according to set steps, track the state transition of each task, and execute user-defined retry logic if necessary to ensure the smooth completion of the workflow.

Free function calculation, pay by quantity, function calculation has a large amount of free.

Free function workflow, pay by quantity, function workflow has a large amount of free.

Function calculation can reliably execute any logic, which can be any video processing operation using FFmpeg, or update video meta data to the database and so on.

The function workflow arranges the corresponding functions, for example, the function of the first step is transcoding, and the function of the second step is to write the corresponding meta database to the database after successful transcoding.

At this point, you should have a preliminary understanding of the custom processing capacity of function computing + function workflow choreography ability to meet almost any of your custom processing needs. Next, this article uses a specific example to demonstrate an elastic and highly available Serverless video processing system based on function computing and function workflow, and compares its performance, cost and engineering efficiency with traditional solutions.

Simple video processing system

Suppose you simply deal with the short video, and the architecture diagram is as follows:

As shown in the figure above, a user uploads a video to OSS, and the OSS trigger automatically triggers the function execution. The function calls FFmpeg to transcode the video, and saves the transcoded video back to OSS.

OSS event trigger, Ali Cloud object storage and function computing are seamlessly integrated. You can set handlers for various types of events, and when the OSS system catches an event of a specified type, it automatically calls the function handler. For example, you can set a function to handle the PutObject event, and when you call OSS PutObject API to upload a video to OSS, the associated function will automatically trigger to process the video.

Simple video processing system example project address

You can deploy your simple audio and video processing system services directly based on the example project, but when you want to deal with large videos (such as test_huge.mov) or perform multiple combined operations on small videos, you will find that the function will fail because of some limitations in the execution environment of function calculation, such as a maximum execution time of 10 minutes and a maximum memory of 3G.

In order to break through the limitation of the execution environment of function calculation, the function workflow FnF is introduced to orchestrate the function to realize a powerful full-function video processing system.

Full-function video processing system

As shown in the figure above, suppose the user uploads a video in mov format to the OSS,OSS trigger to automatically trigger the function execution, and the function calls FnF to extract the audio file in parallel, while transcoding the avi,mp4,flv format. So you can achieve the following requirements:

A video file can be transcoded to various formats and other custom processes at the same time, such as adding watermark processing or updating information to the database in after-process. When multiple files are uploaded to OSS at the same time, the function calculation will automatically scale and process multiple files in parallel.

For each video, it is sliced first, then transcoded in parallel, and finally synthesized. By setting a reasonable slicing time, the transcoding speed of a larger video can be greatly accelerated.

The so-called video slicing is to divide the video stream into a series of sliced files at specified time intervals, and generate an index file to record the information of the sliced files.

Combined with NAS + video slicing, the transcoding of super-large video (larger than 3G) can be solved.

Example project address of full-function video processing system

Example effect:

Function calculation + function Workflow Serverless solution VS traditional solution excellent engineering efficiency self-built service function calculation + function workflow Serverless infrastructure requires users to purchase and manage non-development efficiency in addition to the necessary business logic development, you need to establish the same online running environment. A series of problems, including the installation of related software, service configuration, security updates, etc., only need to focus on the development of business logic, cooperate with FUN tools to arrange resources and deploy parallelism & distributed video processing requires strong development ability and perfect monitoring system to ensure stability. Parallel processing of multiple videos and distributed processing of a single large video can be realized through FnF resource scheduling. Stability and monitoring are handed over to the cloud platform to learn the starting cost. In addition to programming language development capabilities and familiarity with FFmpeg, you may use K8S or flexible scaling (ESS). You need to know more about the meaning of products, nouns and parameters. You can write function codes in the corresponding language and be familiar with FFmpeg use. The launch cycle of the project costs a lot of time and manpower outside the specific business logic. It is conservatively estimated to be about 30 man-days. Including hardware procurement, software and environment configuration, system development, testing, monitoring and alarm, grayscale release system, etc. Expected 3 man-days, development and debugging (2 man-days) + stress test observation (1 man-day) free operation and maintenance, excellent performance self-built service function calculation + function workflow Serverless elastic high availability need self-built load balancing (SLB), elastic scaling The speed of capacity expansion and reduction is slower than that of FC. The inherent millisecond level of FC system is flexible, and the underlying capacity can be expanded quickly to cope with peak pressure. No operation and maintenance is required. Pressure testing of full-function video processing system (FnF + FC) Excellent performance. For more information, please see the transcoding performance table below, monitor alarm query ECS or container-level metrics to provide more fine-grained FnF process execution and function execution. At the same time, you can query the latency and log of each function execution, thus providing a more perfect alarm monitoring mechanism.

Function calculation + function Workflow Serverless Scheme Transcoding performance Table

The experimental video is 89s mov file 4K video: 4K.mov. It takes 188s for cloud service to transcode mov-> mp4. Record this reference time as T

Video slicing time FC transcoding time performance acceleration percentage 45s160s117.5% 25s100s188% 15s70s268.6% 10s45s417.8% 5s35s537.1%

Percentage of performance acceleration = T / FC transcoding time

As can be seen from the above table, the shorter the video slicing time is, the shorter the video transcoding time is, and the function calculation can automatically schedule more computing resources to complete the video transcoding together, and the transcoding performance is excellent.

For lower costs, video processing scenarios with obvious peaks and troughs (for example, video processing requests only in part of the time, and few or no video processing requests at other times), choose to pay on demand and only pay for the computing resources actually used.

Video processing scenarios with no obvious peaks and troughs can be prepaid (annual or monthly plan), and the cost is still very competitive.

Function calculation cost optimization best practice document.

Suppose there is a video transcoding service built based on ECS. Because it is CPU-intensive, the aPCge CPU utilization is used as the core reference indicator to evaluate the cost. Take the total computing power of 10 C5 ECS as an example in an one-month cycle, and the total amount of computation is about 30%. In the scenario, the CPU resource utilization of the two solutions is roughly as follows:

The following billing model is estimated from the above figure:

Prepaid 3CU for function calculation: 246.27 CNY per month. The computing power is equivalent to ECS computing C5ECS computing C5 (2vCPUPower4GB) + cloud disk: the percentage of postpaid payment for monthly function calculation to the total amount of calculation = 80% 998 (246.27 × 3 + 259.2)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.