Shulou(Shulou.com)06/03 Report--

Doing audio and video development for many years, I have shared many blog articles, such as "FFmpeg Tips" series released a few years ago, "Android audio development" series, "live troubleshooting" series and so on. Recently, I want to share my experience in developing and optimizing the player over the years, and also consider opening up the ffmpeg-based player kernel I developed in my spare time, hoping to help beginners in the audio and video field. The content to be launched in the first issue of the article is mainly related to several technical points at the core of the player. The approximate directory is as follows:

1. Player Technology sharing (1): architecture Design

two。 Player technology sharing (2): buffer management

3. Player technology sharing (3): audio and picture synchronization

4. Player technology sharing (4): opening time

5. Player technology sharing (5): delay optimization

This is the second article in a series that focuses on the buffer management of players.

1 Overview

In the previous article, we mentioned the use of buffers to transform the data flow of a single-threaded model into a multithreaded model, which can effectively resist the jitter of network and decoding, prevent frequent stutters, and make full use of the computing power of multi-core CPU, as shown in the following figure:

The player's reading thread puts the "undecoded" audio and video packets output by IO and Parser modules into the "frame buffer" queue, and stores the decoded data into the "display buffer" queue.

2 the function of buffer zone

Let's dig deeper, what is the role of this "frame buffer" and "display buffer"?

2.1 frame buffer

The framebuffer, as a buffer pool between the "reader thread" and the "decoding thread", plays three main roles:

Resist network jitter

Resist decoding jitter

Avoid blurred screen caused by passive frame loss

Assuming that there is no "framebuffer", that is, IO-> Parser-> Decoder, the entire process is serial, then there are the following potential problems:

When the IO network jitter (for example: temporary congestion, unable to read data), then the entire data chain will be stuck, Decoder can only wait for IO to recover

Decoder will also have "jitter", because decoding some complex video frames will take a long time, and if the Decoder is stuck, the IO module will have to wait.

Because the whole process is serial, and each frame must be IO-> Parser-> Decoder to read and process the next frame, then, when the network jitters, the server's TCP protocol stack caches more data. When the network resumes, when it is sent to the client, because the receiving is not timely, the TCP sending queue is full and the frame is passively lost, which leads to decoding blurred screen later because the data is incomplete.

2.2 display buffer

The display buffer, as a buffer pool between the Decoding Thread and the display Thread, serves three main functions:

The necessary conditions for realizing the synchronization of sound and painting

Resist rendering jitter

Assuming that there is no "display buffer", that is, Decoder-> Renderer, the entire process is serial, there are the following potential problems:

Both video frames and audio data are sent to the rendering module immediately after decoding, and the logical processing of audio and picture synchronization cannot be added.

If there is "jitter" in the rendering module, it will block × × directly and cannot decode the data in the frame buffer asynchronously, thus reducing the efficiency.

3 "active buffer" and "passive buffer"

Now that we understand the function of the buffer, let's take a look at how the buffer data is populated.

The data filling of the buffer is mainly divided into two cases, the first is called "active buffer", and the other is called "passive buffer".

Active buffering: means that the player actively pauses the data consumption in the buffer, waiting for the data producer to gradually fill the data, and then resumes it until a certain condition is reached.

Passive buffering: means that the consumption speed of the data can not catch up with the production speed, thus passively retaining the data in the buffer zone.

Active buffering is mostly used in VOD scenarios. In order to reduce frequent stutters, the data will be actively buffering for a period of time (for example, 10s) before the video is played, and then the playback will begin. When the data in the buffer is consumed due to network jitter and other reasons, the buffering will be started again, and so on.

As shown in the figure, assuming that the data in the player buffer is lower than the Low water level, playback will be actively paused and the buffering process will be started until the data in the buffer reaches the M water level.

Passive buffering often occurs in LVB scenarios for two reasons:

The decoding performance of devices such as mobile phones is insufficient, such as soft decoding 1080p high-definition video, resulting in video decoding and rendering speed can not keep up with the video reading speed, resulting in data accumulation in the "frame buffer"

Due to the frequent jitter of the network, the client is unable to get the data in time for decoding and rendering. When the network is restored, the data will be sent down quickly, but the player can no longer consume it quickly (because the playback rate is fixed. Unless the logic of frame tracking is added, the following articles will describe in detail)

4 how to determine the size of the buffer zone?

Now that you understand the role of buffers, how to determine the size of these two buffers? First of all, we need to know what these two buffer sizes affect or determine.

The larger the buffer, the stronger the anti-jitter ability.

The larger the buffer, the higher the memory footprint.

The larger the buffer, the greater the playback delay.

Thus it can be seen that the buffer is not the bigger the better, which needs to be decided according to the actual usage scenario.

The "display buffer" is actually a bridge between the decoding thread and the rendering thread. Because the jitter between decoding and rendering is not frequent, it does not need a particularly large buffer, a minimum of about 3 frames, one frame in production, one frame in consumption, and one frame on standby in the buffer.

While the "frame buffer" is used to resist network jitter, the network jitter is often more frequent, and the jitter time is sometimes longer, so the "frame buffer" should be relatively larger, but on the premise that it does not affect memory and playback delay too much.

For live streaming scenarios, in order to prevent "passive frame loss", the "frame buffer" is often set to "infinite" by default. When it is detected that the buffer reaches a certain threshold, some methods such as active frame loss or multiple playback are started to quickly consume the buffered content, thereby reducing memory and latency.

5 when will the buffer be emptied actively?

In the following scenarios, the player actively clears the data in the buffer:

Player reset

The playback progress bar is dragged

Eliminate cumulative delay

System memory alarm

6 buffer management of IJKPlayer

Ijkplayer is widely used, so let's take it as an example to see what the real case of player buffers is.

After the player is turned on, buffer the 100ms and start playing again.

If stutter is encountered (the buffer is empty), pause playback, buffer to 1000ms, and then start playback.

3. If you encounter stutter again, buffer to 2000ms and replay, and so on, until 5000ms

As you can see, the maximum threshold of its buffer is gradually increased, which is a great user experience optimization, because if the user network is not so bad, you don't have to wait for 5s for the first buffer.

7 Summary

So much for the buffer management of the player. If you have any questions, you are welcome to write to lujun.hust@gmail.com to communicate with us. In addition, you are welcome to follow my Sina Weibo @ Lu _ Jun or Wechat official account @ Jhuster for the latest articles and information.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.