Shulou(Shulou.com)06/03 Report--

This article mainly explains "what is the processing flow of WebRTC Audio before entering Encoder". The explanation content in this article is simple and clear, easy to learn and understand. Please follow the ideas of Xiaobian slowly and deeply to study and learn together "what is the processing flow of WebRTC Audio before entering Encoder"!

In WebRTC, there are two major areas of Audio data that require special attention before it is sent to the encoder: data acquisition and Audio Processing.

data acquisition

Data acquisition is mainly processed by Audio Device module, and it is platform and configuration dependent. For example:

Mac computers, using the CoreAudio API, generally use the default built-in sound parameters fs=48kHz, stero.

Windows computers, WASAPI in WebRTC. According to different sound card parameters, sampling rate and other parameters can be selected more, for example, some computers builtInAEC open, fs=16kHz, Mono, if the sound card Audio Enhancement is closed, then the output fs=48kHz, stero.

Android generally uses the java layer AudioRecord framework.

iOS generally uses the AudioUnit framework.

In addition, the data acquisition part also involves USB headphones, 3.5mm headphones, Bluetooth headphones and other peripherals. These devices also have an impact on the subsequent Audio Processing on the audio link, such as adding delay for Audio acquisition, headphones with Speech Enhancement processing will modify the audio spectrum, and improper use of some headset peripherals may cause no sound on the audio link.

Audio Processing

Audio Processing mainly includes AEC, AGC, NS, etc.:

AEC---Acoustic Echo Cancellation.

AGC---Automatic Gain Control, which is used to adjust the volume of the input signal.

NS---Noise Suppression.

Data output from Audio Devices passes through audio processing modules such as AEC, NS, AGC, etc. in turn.

1. AECAEC algorithm selection

In WebRTC, AEC has 4 optional algorithms:

builtInAEC, usually Windows, Android system, builtInAEC will be enabled by default.

AECM, Echo Cancellation Algorithm for Mobile, for Android and iOS.

AEC algorithm, echo cancellation algorithm for Windows/Mac Desktop. Of course, AEC can also be used on the mobile side. In some cases, the echo leakage performance is better than AECM. However, the latest WebRTC has removed the old AEC code.

AEC3 algorithm, Google's revision of the old AEC algorithm, AEC3 has completely replaced the old AEC algorithm.

Under normal circumstances, only one of these four AEC algorithms can be selected, otherwise multiple AEC will be done, and the damage to sound will also increase. In the case of last resort, 2 AEC may be used, for example, Windows computer, buildInAEC cannot be turned off and the effect is poor, AEC3 must be turned on, then 2 AEC are used.

aec_dump

During a call, use StartAecDump to enable aec_dump function, aec_dump will record 3 files, one is input.wav that has not entered the Audio Processing module, one is the output file reverse.wav of Speaker Render, and one is ref_out.wav that has been processed by Audio Processing.

Normally, input.wav - reverse.wav = ref_out.wav.

These three files can be used to analyze whether there is a problem with the echo cancellation algorithm.

2.AGC

WebRTC AGC has two algorithms:

Legacy AGC

AGC2

Here is a brief overview of Legacy AGC performance. original speech

AGC voice, Legacy AGC itself has no noise reduction function, noise and voice at the same time amplified.

3.NS

At present, WebRTC Noise Suppression module can filter out relatively stable background noise, such as white noise, air conditioning sound, etc. However, NS module is ineffective against very loud background noise and babble noise, which is also one of the factors of "noisy and unclear" in the case of Microphone amplifier (of course, there are other reasons for noise, such as AEC performance, etc.).

White noise recording

White noise is filtered out.

4. Other enhancement algorithms

High Pass Filter, used to filter low-frequency noise, for example, we can filter out low-frequency noise below 100Hz or even 200Hz.

Typing Detector, can filter out the sound of keyboard typing.

Residual Echo Detector.

optimization point

Acquisition (of course, including playback) is prone to no sound problem, so it is necessary to optimize the sound card adaptation of the acquisition (sound card driver) end.

AEC distinguishes platforms: A. Windows platform, general Windows sound board inside there is a "Audio Enhancement", which contains builtInAEC, the default is open, through the Windows API to close this Audio Enhancement is almost impossible. Some builtInAEC effects are poor, and AEC3 must be turned on. Typically, builtInAEC and AEC3 are turned on at the same time to minimize echo. B. Mac computers do not have buildInAEC by default, and usually use AEC3 directly.

Add "near talk,""far talk,""double talk," and "no talk" states to the AEC algorithm. Combine these states and take different operations. For example, when there is only far talk, AGC is not performed, which can reduce the near-end noise.

Thank you for reading, the above is "WebRTC Audio before entering the Encoder processing flow is what" the content, after the study of this article, I believe that everyone on WebRTC Audio before entering the Encoder processing flow is what this problem has a deeper understanding, the specific use of the situation also needs to be verified by practice. Here is, Xiaobian will push more articles related to knowledge points for everyone, welcome to pay attention!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.