Shulou(Shulou.com)11/24 Report--

Recently, at the Techo Frontier Technology Forum of Tencent Global Digital Ecology Conference, Dr. Liu Shan, Tencent Distinguished scientist, Vice President of Tencent Cloud, General Manager of Tencent Multimedia Lab and Deputy General Manager of Tencent Video's Intelligent creation and content platform Department, gathered with Professor Tao Xiaoming, Professor of Electronic Engineering Department of Tsinghua University and winner of the 2021 Science Exploration Award in the field of information electronics. This paper deeply discusses the cross-cooperation in the field of semantic communication and video codec.

Dr. Liu Shan has been committed to the technical research of multimedia and related fields, including signal and information processing, audio and video and spatial media data compression, transmission interaction and intelligent applications. The Tencent Multimedia Lab, which she leads, mainly involves two major aspects: the exploration and standard setting of cutting-edge technologies, and the landing of product-oriented technology research and development and applications. Professor Tao Xiaoming focuses on semantic communication in wide-area specific scenarios, and solves the pressure of large-capacity multimedia services on wireless network bandwidth demand by integrating human brain visual perception and cognitive mechanism into the process of network transmission and communication.

Combining their expertise in their respective fields, the two experts conducted in-depth discussions on three aspects: brain-inspired video quality evaluation, semantic enabling video codec, and cross-domain cooperation between semantic communication and video codec.

A new idea of multimedia quality evaluation can introduce the characteristics of human brain perception, cognition, prior knowledge and so on.

Dr. Liu Shan mentioned the landing of product-oriented technology research and development and application in Tencent Multimedia Lab, which can be divided into three major directions in terms of technology segmentation: media compression and transmission, intelligent fusion media, and interactive immersive media. These directions are closely related to current hot concepts such as AIGC, XR and meta-universe. She stressed that multimedia is a system, including signal processing, compression, transmission, interaction, rendering and modeling, which requires joint optimization to achieve the best performance and user experience. Performance and user experience need an efficient quality evaluation system for quantitative evaluation. It is a very meaningful innovation and exploration to explore and learn from the characteristics of the brain on the traditional quality evaluation system to complete the quality evaluation of multimedia.

Professor Tao Xiaoming believes that there are three characteristics of the brain that can be related to multimedia communication. The first is perception, the human brain can subjectively and qualitatively judge QoE, and can directly perceive whether it is good or not; the second is in cognition, the ability of global search and reasoning of the human brain, if it can be introduced into the coding and decoding of communications, on the one hand, it can reduce the complexity of video coding, but also better protect context-important semantic information in the process of transmission. Third, in terms of prior knowledge, the brain can automatically match the cognition that has been touched before, and if applied to communication, it can reduce the demand for bandwidth in some special scenarios.

Methods such as deep learning and machine vision can solve the coding and decoding requirements in more general and special scenarios.

Video codec is particularly important in the vigorous development of 5G and even 6G, especially in multimedia data compression. Audio, video, images and emerging VR, high-dimensional data, etc., usually have a large amount of data, which requires a lot of storage space and transmission bandwidth. In order to solve this problem, video codec technology arises at the historic moment, after the development of several generations of standards, such as H.264 / AVC, H.265 / HEVC, H.266 / VVC and so on. Deep learning has made some progress in audio signal compression, but it is still challenging in video signal compression.

Dr. Liu Shan pointed out that in machine vision, information (such as voice and image) processing, deep learning and artificial intelligence have played a role in many practical applications, thus promoting the exploration of use in video codec. At present, in the formulation of video codec standards, Tencent Multimedia Lab has also found many technical proposals and trends to meet the needs of different applications and environments.

Professor Tao Xiaoming also said that in special situations such as rural areas, left-behind elderly and children, we can introduce the idea of brain science and use EEG analysis to extract people's subjective perception, so as to improve the user experience. In addition, Professor Tao Xiaoming also introduced a coding and decoding method based on spatio-temporal sketch map, which can reduce the amount of data by extracting video features such as outline, semantics and relations. Therefore, at the receiving end, we need to use generative machine learning and reinforcement learning methods, which can reduce the amount of data transmission under the special Yangtze River, in order to meet the needs of users and generate videos with optimal user experience.

Semantic communication and video codec, or cross-domain cooperation can be achieved.

Dr. Liu Shan believes that quality evaluation is omnipresent, including the 3D video compression transmission currently being studied by Tencent Multimedia Studio. She believes that these areas are not yet mature, there is a lot of room for exploration, and the research methods based on human brain feedback have great potential, which may promote the improvement of multimedia codec standards in the future. Tao Xiaoming added that in the fields of AR, VR and games, EEG signals can provide valuable information about user experience, such as interaction, feeling and delay, which is also a new dimension for the study of semantic communication. I look forward to working with Tencent Multimedia Lab to understand more user needs in the future.

Tencent Multimedia Lab has participated in international standard-setting on behalf of Tencent since the beginning of 2018. so far, more than 800 technical proposals have been adopted by a number of international standards and more than 1500 authorized patents have been accumulated. dozens of people have played important roles in the international standard-setting process, and their technical contributions have been widely recognized by international standards organizations and the industry. The laboratory won ISO / IEC Outstanding contribution Award, AVS Industrial Technology Innovation Unit Award, Technology and Engineering Emmy Award (Technical Emmy Award), Technology Lumiere Award (Technology Lumiere Award), leading Science and Technology Achievement Award of Digital Expo, World artificial Intelligence Conference "Town Hall Treasure". At the same time, the research and development of multimedia core technology is applied to a variety of Tencent products to provide quality services for 100 million users. Since 2018, we have invested in the research and development of immersive media XR technology, system construction and AIGC capacity intelligent content production, including VR. VR was first applied to Tencent products in 2019, and then successively provided technical support to Xinhua News Agency, the Imperial Palace, Dunhuang and other cooperative projects as well as Tencent WE Conference, Tencent Global Digital Ecology Conference and Siberian Tiger National Park. General solutions such as "VR Panorama", "Free Visual Angle", "Point Cloud Modeling" and "Point Cloud Compression" with multimedia laboratory technology as the core have been launched on Tencent Cloud's official website. In 2019, Dr. Liu Shan, an outstanding scientist from Tencent, proposed to the multimedia lab team led by him to carry out the research and development of "intelligent content production" technology. and in the following time, he led the team to build a number of core technologies and gradually improve the capability matrix, which was applied to a number of content production and creation business scenarios. The 2023 team product XMusic was awarded the "Treasure of the Town Pavilion" of the 2023 World artificial Intelligence Conference. In the future, multimedia laboratories will continue to invest in related technology construction, and continue to make low-level technical input for the construction of to B industry scenes such as education, industry, literature and tourism, real estate and home furnishings, etc.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.