Shulou(Shulou.com)06/03 Report--

I have been engaged in audio and video development for many years and have shared many blog posts, such as FFmpeg Tips series released in previous years, Android Audio Development series, Live Trouble Shooting series, etc. Recently, I want to share my experience in developing and optimizing players over the years, hoping to help beginners in the audio and video field. The first issue of the article to launch the content mainly involves the player core of several technical points, the approximate directory is as follows:

1. Player Technology Sharing (1): Architecture Design

2. Player Technology Sharing (2): Buffer Management

3. Player Technology Sharing (3): Sound and Picture Synchronization

4. Player Technology Sharing (4): First Opening Time

5. Player Technology Sharing (5): Delay Optimization

This article is the third in a series of articles, focusing on talking about the player's audio and picture synchronization.

1 summarizes

First of all, we give the definition of sound-picture synchronization:

Sound and picture synchronization means that every frame of picture being rendered by the player and every sound being played are strictly corresponding, and there is no deviation that can be distinguished by human ears and naked eyes.

2 Time stamps

The machine has no eyes or ears, so how does it perceive that a frame of video data in a video file belongs to the same time as a frame of audio data?

In real life, the way to tell if two things happen at the same time is to record and compare the date and time of the event. Similarly, we can mark the audio and video data with a "timestamp" when it is "produced", representing its birth time. From this, the player knows how to judge.

As shown in the figure, when audio and video are collected (produced), each frame is marked with a "timestamp"(note that the timestamp of audio and video needs to use a common timestamp standard). Assuming that the absolute time point of the system at the beginning of audio collection is taken as the timestamp 0, assuming that audio is sampled for 20ms and video frame rate is 30fps ->, i.e. 33ms, a frame is generated, then the approximate increasing rule of the timestamp of video frame and audio frame is as follows:

Video frames: 10, 43, 76, 109, 142,…

Audio frames: 0, 20, 40, 60, 80, 120,…

Note: In practice, data acquisition will not be so "uniform" and will fluctuate slightly.

In this way, the player can perceive the "relative relationship" between the audio frame and the video frame through the timestamp information attached to the audio and video frames, and the synchronization of sound and picture is easy. The judgment method is as follows:

The method of synchronization of sound and picture: judge whether the time stamp difference between video and audio frames is within a certain "threshold" range, if so, render/play, otherwise "wait" to the appropriate time.

3."Threshold" for synchronization of sound and picture

Since audio and video are captured separately, their timestamps cannot be exactly matched, so they can be considered synchronous as long as they are within a reasonable "threshold" range. So how is this threshold defined?

As shown in the figure, this "threshold" is defined by an international standard called RFC-1359, which is defined as follows:

1. Undetectable: audio and video timestamp difference between: -100ms ~ +25ms

2. Can detect: audio lag more than 100ms, or more than 25ms ahead

3. Unacceptable: Audio lagging more than 185ms or leading more than 90ms

With this international standard and experience value, we don't have to guess the threshold value for sound and picture synchronization ~~

4. Realization of synchronization of sound and picture

There are generally three strategies for achieving synchronization of sound and picture:

1. Video sync to audio

2. Audio sync to video

3. Audio and video synchronization to external clock

A common strategy is to synchronize video to audio.

Because the audio is streaming, according to the regular constant speed to play, can appear more "smooth," and video playback is actually a picture refresh display, its refresh time adjustment is relatively easier, the user's naked eye sensitivity is also weaker.

Here's how to synchronize the player's sound and picture using the "video sync to audio" approach:

As shown in the figure, A represents the audio frame, P represents the video frame, the audio frame is continuously sent to the speaker to play at a predetermined rate, and each time a frame of audio data is played, the timestamp of the audio frame is updated to Master Clock as the master clock, and the video frame refers to Master Clock to decide whether to render, when to render or discard it. The algorithm is as follows:

1. Assuming min is the audio and picture synchronization threshold (e.g., 25ms), the current video timestamp can be sent to rendering if the absolute difference between it and master clock is within the threshold range of 25ms.

2. If diff > 0 (pts > m-clock), the video frame is ready in advance. In this case, sleep can be used to wait for the master clock. However, in order to prevent the sleep time from being too long due to the abnormal timestamp, a max abnormal threshold (e.g., 1000ms) can be set. If diff exceeds this max, it can be considered as an abnormal frame and discarded.

3. If diff < 0, that is pts < m-clock, then the video frame lags. In this case, video frames that lag beyond a predetermined threshold (e.g., 25ms) should be discarded

The core idea of sound-picture synchronization is here. It doesn't seem to be that complicated, right?

6 summarizes

Player sound and picture synchronization, share here, if you have questions, welcome to write lujun. gmail.com communication. In addition, you are also welcome to pay attention to my Sina Weibo @ Lu_Jun or Weixin Official Accounts @Jhuster for the latest articles and information.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.