Shulou(Shulou.com)11/24 Report--

"Zhi Zhen Picture quality" is a core capability of Sound Network's "real-time HD super quality" solution. It not only supports 1080p and 4K HD picture quality on mobile, but also enhances and improves picture quality through algorithms such as end-to-side real-time super division, dark light enhancement and color enhancement. Compared with the technologies with higher technical threshold, such as super division and dark light enhancement, it is not simple to support 1080p and 4K high-definition picture quality in real-time interaction. There are also a series of technical challenges behind it.

Overall, the technical challenges of supporting UHD video in real-time interaction can be divided into three challenges: "network transmission of large amounts of data and weak network countermeasures", "processing performance and frame rate requirements of all aspects of the link", "integrated device availability and QoE experience issues". Next, we will analyze the sound network best practices behind each technical challenge one by one.

The huge amount of audio and video data tests the ability of network transmission and weak network confrontation.

In daily life, we generally need faster network bandwidth to download or watch higher-definition video, and this is also the case in real-time interaction. In order to achieve the transmission of ultra-high-definition video such as 4K, we generally need 10m, 20m or even higher bandwidth to achieve stable video transmission. Under ordinary network conditions, it is easy to cause a large number of network packet loss or jitter, causing video stutters.

The higher the definition of the video, the larger the amount of video data, so audio and video coding technology is usually used before transmission to compress the size of audio and video data, making it easier to store and transmit audio and video. However, the amount of compressed video data still tests the transmission ability of the network. In order to solve the network quality problem in real-time interactive scenarios, we need to rely on the transmission protocol, weak network countermeasure algorithm and media transmission strategy.

In UHD video transmission scenarios, in addition to using efficient coding and compression technologies such as PVC, H265 and B frames, the sound network can make video calls smooth under packet drop in 4K picture quality scenarios, thanks to the self-developed transmission protocol AUT, anti-packet loss FEC algorithm and adaptive media transmission strategy.

On the one hand, the AUT transport protocol developed by Acoustic Network adopts more reasonable transmission architecture and better algorithm, which brings greater transmission capacity and higher packet loss resistance boundary. At the same time, it also supports Scalability (scalability) of multi-person video. Many problems will be magnified in UHD video scenarios, including weak network countermeasures. The originally good network bandwidth can accept 720p image quality, but it is not enough under 1080p and 4K picture quality. At this time, the most appropriate bit stream will be distributed according to the network quality of each receiver, for example, from 4K 60FPS to 4K 30FPS, or even 1080p, so that each receiver can get a smooth experience that matches its own network status.

On the other hand, the FEC algorithm developed by the acoustic network can improve the packet loss recovery rate, reduce the decoding delay, and finally improve the stutter rate and decoding delay time under the weak network. At the same time, the weak network countermeasure algorithm of the acoustic network, combined with the hybrid ARQ (automatic retransmission request) media transmission strategy, can realize an adaptive weak network countermeasure system according to multi-dimensional input parameter information, network environment, user scene and result feedback, ensuring the ability to resist weak network in high-resolution scenes and improving video quality (clarity, fluency, delay).

Processing performance and frame rate requirements of all links in the whole link

For high-resolution and high-frame-rate video scenes, in addition to coding transmission and weak network countermeasure capabilities, the frame rate performance of video capture and rendering and the CPU performance overhead of uplink and uplink processing are often bottlenecks. Because the amount of memory data processed per second can be up to hundreds of MB or even more than 1GB, so for 2K / 4K 60FPS scene sound network, we do all-link depth optimization, such as collection, rendering, hardware codec and so on. The whole platform supports zero-copy link, making full use of the hardware acceleration processing capability of GPU, and reducing video data handling operations as much as possible, so as to reduce the consumption of CPU for a large number of video data processing and transfer.

This is like using containers instead of traditional bulk cargo loading and unloading, which can greatly improve the efficiency of cargo circulation. In addition, by dividing the whole video processing link into multiple sub-tasks, which is similar to the multi-level pipeline of modern factories, the parallelism of video processing can be improved, and finally the throughput of the whole video link can be improved.

At the same time, for 4K 60FPS UHD screen sharing acquisition, the sound network realizes 60FPS full frame rate acquisition through accurate timing control, while using the system's native data format to avoid additional data copying and conversion, for example, screen acquisition directly outputs BGRA format CVPixelBuffer on Mac platform and does not need additional format conversion.

(note: CVPixelBuffer: the core video pixel buffer refers to the image buffer that holds pixels in the main memory. CVPixelBuffer can be used by applications that generate frames, compress or decompress video, or use Core Image. )

In the rendering of 4K 60FPS high frame rate video, Sound Network uses the VSync mechanism displayed by various platforms to design a high frame rate rendering system to avoid high frame rate video stream losing frames in the rendering module, at the same time, it makes the rendering of high frame rate scenes more uniform, and achieves the effect of end-to-end 2K / 4K 60FPS stable frame on middle and high-end devices. (note: VSync is the abbreviation of vertical synchronization. The basic principle is to synchronize the FPS frame rate of the video with the refresh rate of the monitor in order to avoid the phenomenon of picture "tearing".

Integrated device availability and QoE experience issu

In practical application scenarios, UHD video inevitably has to face a lot of device compatibility and experience problems. Compared with ordinary SD and HD video, UHD video is more likely to encounter problems that the device does not support, or the device supports but obviously stutters or heats up. Especially in the multi-person video interaction scene, the equipment and network conditions are complex, the sender can achieve 4K 60FPS, but the equipment quality of the receiver audience is uneven. For example, some devices can not support 4K 60FPS video decoding, some decoding can only solve more than 20 frames, the video frame is uneven, resulting in video not smooth, these are device availability issues, but also become a stumbling block to the landing of business applications.

To solve the above problems, the solution of acoustic network can essentially be summarized as Scalability (scalability) based on device capabilities and network conditions, making comprehensive use of a variety of tools and adaptively adjusting the parameter configuration and strategy of the engine, such as the sender can adaptively choose the most appropriate resolution, frame rate and code rate, etc., provide multi-level service capacity and flexibility, combined with some scene API, give reference practice in different scenarios. Maximize the overall availability and QoE experience requirements of the business.

For example, the AutoAdjust (automatic adjustment) adaptive strategy adopted by the sound network can synthesize the type of service, the performance of the equipment, the network conditions and the status of each processing module of the link, and adaptively select the most suitable resolution, frame rate and bit rate, as well as the gear, software and hardware coding of the video processing module and the configuration of network policy parameters, and so on, so as to ensure the experience of video quality as far as possible on the premise of avoiding equipment heating and jam.

The solution of the sound network also has equipment classification and equipment capability query, using a normalized way to define how much the equipment is capable and how much content it can support. At the same time, you can query whether the device has the decoding capability of 4K 60FPS, so as to customize the best solution according to the business scenario.

It is with the continuous practice of the above technologies that the sound network can more perfectly support 1080p and 4K ultra-high definition picture quality in real-time interaction, thus further realizing the ability of "achieving picture quality", such as video picture quality enhancement and picture visual effect improvement. in addition, sound network "real-time high-definition super picture quality" also includes eight gift packages of beauty, silky fluency, low-code HD, PC broadcasting, game upgrade, data monitoring and use. Help developers and enterprises to achieve a comprehensive upgrade of video picture quality, user experience and interactive play, and expand a broader space for revenue growth.

If you want to learn more about Soundnet's "Real-time HD Super Picture quality" solution, you can find this article on the official account, click below at the bottom of the article to read the original text for further consultation.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.