Shulou(Shulou.com)06/03 Report--

Doing audio and video development for many years, I have shared many blog articles, such as "FFmpeg Tips" series released a few years ago, "Android audio development" series, "live troubleshooting" series and so on. Recently, I would like to share my experience in developing and optimizing players over the years, hoping to help beginners in the field of audio and video. The content to be launched in the first issue of the article is mainly related to several technical points at the core of the player. The approximate directory is as follows:

1. Player Technology sharing (1): architecture Design

two。 Player technology sharing (2): buffer management

3. Player technology sharing (3): audio and picture synchronization

4. Player technology sharing (4): opening time

5. Player technology sharing (5): delay optimization

This is the fifth in a series of articles, mainly about how to optimize playback delay. Since the playback delay needs to be jointly optimized from each link of "Video production-> Distribution-> playback", it cannot be solved by the player alone. Therefore, this article will introduce some of the overall reasons for delay and optimization ideas, rather than how to solve the delay problem by the player itself.

Broadcasting is generally divided into two types, one is live streaming, such as live broadcast of online celebrities, live events, summit meetings, security camera surveillance, etc., and the other is online on-demand video files, such as online movies and TV dramas, education and training videos, live playback and so on. It is only in the face of live scenes that there is a demand to reduce the delay.

1 measurement of delay

To optimize the playback delay, you must first know how to measure the delay. There are many methods in the industry, which are briefly introduced as follows:

1.1 online stopwatch

The VJ faces the Beijing time / stopwatch, and then compares the difference between the time displayed in the player screen and the Beijing time / stopwatch, which is the most commonly used measurement method.

1.2 SEI Fram

The full name of SEI is supplementary enhancement information (Supplemental Enhancement Information), which provides a way to add additional information to the video bitstream. It is one of the features of video compression standards such as H.264/H.265.

As shown in the figure, we can regularly insert some SEI frames on the streaming media server (or VJ side) to record the current NTP standard time. When the player receives the SEI frame, the time of the SEI frame is extracted and compared with the current system time to get the delay difference.

1.3 Audio waveforms

As shown in the figure, the host side and the player side start to write audio data into the file at the same time, and then compare the audio waveforms of the two files to measure the delay difference. This "same moment" can be notified by message, a common NTP time can be agreed, or the trigger can be triggered by SEI frame.

2 how does the delay occur?

There are many factors that affect the playback delay. Here are a few:

1. Bandwidth and transmission distance

two。 Network jitter and congestion control

3. Server-side GOP cache

4. The buffer zone of each link

5. Slice size of HLS protocol

6. Data processing performance: such as beauty filter, clipping stretching, encoding and decoding, video rendering, etc.

2.1 Network bandwidth and transmission distance

This is a basic concept, the impact of bandwidth and transmission distance on the delay is very important, especially the domestic cross-provincial and cross-operator export bandwidth restrictions and other reasons, will lead to greater delay. This is why the concepts of edge storage and edge computing have been mentioned a lot in recent years, especially in intelligent security surveillance scenes, since most of the consumption of surveillance video is in local areas, if the video storage and multimedia / intelligent processing are put on the edge, the efficiency and experience can be greatly improved.

2.2 Network jitter and congestion control

Streaming media transmission based on TCP protocol is very unfriendly to delay, because many characteristics of TCP protocol inevitably increase the delay, such as three-way handshake, packet loss retransmission, some congestion control strategies, etc., so the delay will be larger under the weak network with frequent jitter.

2.3 GOP cach

As shown in the figure, when we talked about optimization in the previous article, we mentioned the need to cache a GOP data on the server. When the player applies for a bitstream, the cached GOP is quickly distributed to achieve instant access.

However, this piece of GOP data obtained by the player is a piece of historical GOP data, not real-time data. It will take some time to play this GOP cached data. Assuming that the GOP size of the LVB is 5s, it will directly cause the delay of the player to reach at least 5s.

2.4 buffer

Whether it is the push end, the distribution end or the player itself, there will be some buffers to deal with the problems caused by jitter such as network and performance.

Assuming that the frame rate of the video is 30fps, each buffer of 30 frames is equivalent to a delay of 1 second. Thus it can be seen that the delay caused by the buffer is relatively large, which is one of the biggest factors leading to the delay.

2.5 HLS slices

If the HLS protocol is used for playback, as shown in the figure, assuming that the server slices to generate ts files in units of 3 seconds 10 seconds, the playback delay will directly reach 3 seconds 10 seconds. This is a hard defect of this protocol in terms of delay. If it is sensitive to the delay of live broadcasting, RTMP or HTTP-FLV protocol is generally used to pull the live stream.

3 how to optimize the playback delay?

Similar to the idea of opening optimization, the key to optimize the playback delay is to start with the various factors that affect the delay. The basic ideas are listed below.

1. Network line optimization

Select high-quality CDN acceleration services to ensure network bandwidth and line resources for transmission

Speed measurement and route selection, dynamic monitoring, intelligent scheduling to ensure the transmission quality of each stream

two。 Drop buffer

The production side reduces the GOP size, thus reducing the server side GOP cache size

The production end dynamically adjusts the frame rate and bit rate according to the sending buffer.

The player actively loses / chases / plays multiple frames to speed up the consumption of buffers.

3. Protocol / transport optimization

Based on TCP-> Based on UDP, such as QUIC,RTP/RTCP

Transmission strategy optimization: forward error correction, packet loss retransmission strategy, congestion control optimization, key frame request, etc.

4. Processing performance optimization

Hardware codec / beauty filter / rendering, etc.

Algorithm performance tuning

As each optimization here requires a large space to elaborate, there are also many related discussions and articles on the Internet, so I will not start talking about it. I will have the opportunity to share it deeply later.

4 Summary

This is the end of the delay optimization of the player. If you have any questions, you are welcome to write to lujun.hust@gmail.com for communication. In addition, you are welcome to follow my Sina Weibo @ Lu _ Jun or Wechat official account @ Jhuster for the latest articles and information.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.