Shulou(Shulou.com)06/03 Report--

Doing audio and video development for many years, I have shared many blog articles, such as "FFmpeg Tips" series released a few years ago, "Android audio development" series, "live troubleshooting" series and so on. Recently, I would like to share my experience in developing and optimizing players over the years, hoping to help beginners in the field of audio and video. The content to be launched in the first issue of the article is mainly related to several technical points at the core of the player. The approximate directory is as follows:

1. Player Technology sharing (1): architecture Design

two。 Player technology sharing (2): buffer management

3. Player technology sharing (3): audio and picture synchronization

4. Player technology sharing (4): opening time

5. Player technology sharing (5): delay optimization

This is the fourth in a series of articles, mainly about how to optimize the opening time of the player.

1 definition of the first opening time

Opening time: the time from the click to play to the display of the first frame. Generally speaking, the "first screen opening time" means that the player's "first opening time" is less than 1 second.

The opening speed is the simplest and most direct experience for users, so it is usually the focus of optimization in player development.

The following is a typical table of the relationship between the opening speed and the user's experience:

2 influence factors of the first opening time

To optimize the player's opening time, we need to know the factors that affect the player's opening time. The following figure briefly shows the whole process of the player applying to the server to play a video stream.

To optimize the opening time of the player, first pay attention to each link in the playback process. As shown in the figure, the list of possible optimization points is as follows:

Optimize the time to apply for the playback URL address of the resource-> strive to get the URL before the user clicks to play.

DNS parsing optimization-> complete DNS parsing ahead of time and cache the results

Optimization of server connection and data transmission speed-> mainly optimization of network transmission between server node and player

Analysis and optimization of media information of video stream-> mainly the optimization of analytical extraction algorithm

Decoding and rendering strategy optimization-> GOP cache to ensure that the first frame is decoded and rendered for key frames

Other optimization methods-> speed measurement, line selection, decoding algorithm performance, etc.

In this process, there are some factors that will determine the first opening time of the player in each link. We will expand the optimization idea in detail below.

3 first opening optimization method

3.1 optimize the timing of URL acquisition

As shown in the figure, a playlist shows that each video corresponds to a URL playback address of the resource. If APP goes to the backend business server to apply for this URL playback address after the user clicks on the video, it will undoubtedly increase the time-consuming of a HTTP request and response, especially when the network is unstable.

Therefore, this is an optimization point that can be done at the APP layer: when pulling the video playlist, the URL playback address of the video is also pulled down "at the same time". After the user clicks on the video, there is no need to apply for a playback address from the server to start playing immediately.

3.2 optimize DNS parsing time

The URL addresses of video resources often contain domain names, such as:

Http://jhuster.com/video/movie.mp4

Before playing, the player needs to parse the DNS to resolve the jhuster.com domain name to the IP address of a server, and then connect to it through TCP to send resource requests.

We simply measured the resolution time of this domain name on the 17CE website:

As you can see, the average DNS parsing time is about 673ms, but in many areas (such as Foshan Telecom and Beijing Telecom), the parsing time is more than 2 seconds. It is conceivable that the slow DNS parsing in these areas is fatal to the player's opening time.

To ensure that video playback in all regions is only affected by the speed of DNS resolution, in addition to purchasing paid professional DNS resolution services for the domain names of video resources, some optimizations can also be made at the player level, as shown below:

A new DNS result cache module is added inside the player

DNS parsing is performed regularly in the asynchronous thread, and the domain name & IP table is cached in memory

When the video is played, check the table directly, take out the IP address corresponding to the domain name, and send it to the player.

Note:

The number of resource domain names of an APP is convergent, not unlimited, so when APP starts, you can send it to the player to complete DNS parsing and caching in advance.

A domain name that is not configured in advance will still start slowly at the first time of resolution, but the domain name can be taken from the local cache the second time.

You need to pay attention to the refresh of the cached IP address:

There is a TTL timeout for DNS parsing. Remember to refresh the parsing before it expires.

Monitor mobile network handoff events, for example, after WiFi is switched to 4G, the cache needs to be cleared.

For example:

DNS Cache Map:

If you want to play URL: http://jhuster.com/video/movie.mp4

Can be directly replaced with playback: http://185.199.108.153/video/moive.mp4

Where's the pit?

When we really play it with the ip address, we will find that the server will refuse access, for example, the following error is reported:

Reason: the CDN service provider of the video resource needs to know who is applying for the video resource-> for metering and billing, the service provider determines the access based on the "domain name", so using IP directly will encounter the above problems.

How to solve it?

HTTP protocol: there is a HOST field that records the domain name of the server

RTMP protocol: there is a tcUrl parameter that records the full player address (including the domain name of the server).

In fact, CDN service providers do not take the domain name from URL, but "extract" the domain name from the above fields in the HTTP/RTMP protocol. Therefore, we need to modify the underlying HTTP/RTMP code module of the player and provide an interface to enter the original domain name in URL into the above protocol field.

3.3 optimize server connection and data transfer time

The player gets the IP address of the server through DNS parsing, and the next step is to connect to the server through TCP and send a request to read the data. In this process, the connection speed with the server and the data transmission time are very important, which directly affect the first experience.

This factor is not an optimization that can be performed at the player level, so to mention briefly, the key factors for optimization are server load, CDN node distribution, and bandwidth. This is why the average APP company purchases and uses CDN services.

3.4 optimize the reading strategy of media information

What's in the media information of the video?

Whether to include: audio, video

Audio and video coding formats, such as H.264, AAC, etc.

Video information, such as resolution, frame rate, bitrate

Audio information, such as sampling rate, bit width, number of channels

The total duration of the code stream

Other additional information such as author, date, etc.

It can be seen that media information is still very important for initializing the player. Different audio and video packaging formats, media information storage location is not quite the same, such as flv format, media information is often stored directly at the beginning, so it is relatively easy to read the first time. In mp4 format, you will often encounter the situation where moov is at the tail, which is the focus of player optimization, as shown in the figure:

For this kind of mp4 file that stores the media information in the tail, the default player needs to download the entire mp4 to get the media information, which is extremely unfriendly to the first opening.

How to optimize it? -- dual IO technology

As shown in the figure, for the mp4 file with moov media information at the tail, after the player reads certain data, if the player determines that the moov is at the tail, it can pause the thread and start the second thread to read the tail moov through the http range field, so as to get the key media information. The advantages of this technical strategy are as follows:

The video of moov at the tail can be played without downloading the entire mp4.

The downloaded part of the first IO can be used instead of being discarded.

3.5 optimize the parsing time of media information

Where is the workload of media information analysis?

Determine the encapsulation format of the stream, such as mp4,flv,m3u8, etc.

Extract the media information from the data according to the protocol agreement of the encapsulation format

In order to improve the compatibility with non-standard bitstreams, ffmpeg uses a very complex parsing strategy. Even if metadata has been extracted from the bitstream, it will still do a variety of double check. For example, many times try_decode_frame tests whether the data can really be successfully decoded, resulting in the lower speed of the underlying ffmpeg-based players.

How to optimize it? If it is a player based on ffmpeg kernel, the common methods are as follows:

Reduce probesize

Reduce analyzeduration

Audio and video format of preset bitstream

3.6 optimize decoding and rendering of the first frame

We know that the encoded video frame is divided into I frame, B frame and P frame, and the I frame is the key frame, which can decode the image independently, and the Bash P frame is the forward prediction frame / bi-directional reference frame respectively. It is necessary to refer to I frames or before and after frames to decode the image.

Therefore, in order to decode the first frame as soon as possible, it is necessary to ensure that the first frame sent to the encoder is the I frame.

As shown in the figure, in a live broadcast scenario, if the viewer pulls the stream at C time, you can pull an I frame and quickly complete the decoding and playback. However, if you happen to pull the stream at exactly A time point or B time point, you will not be able to decode and wait until the next I frame to complete the decoding rendering.

Therefore, most LVB cloud vendors will cache a GOP data on the server. Whenever the player applies for playback, it will first send such a GOP data that begins with the I frame, thus speeding up the decoding and opening of the player.

As far as the player is concerned, it should be noted that before the first frame is rendered, do not buffer actively, but render the first frame as soon as possible.

4 Summary

Of course, there are many other optimization strategies for player opening time, such as speed measurement and line selection, decoding algorithm performance, etc., which are not widely used, so we will not introduce them here. With regard to the optimization of the opening time of the player, so much for sharing. If you have any questions, you are welcome to write to lujun.hust@gmail.com to communicate. In addition, you are welcome to follow my Sina Weibo @ Lu _ Jun or Wechat official account @ Jhuster for the latest articles and information.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.