Shulou(Shulou.com)06/03 Report--

In the previous article, "WebRTC Development practice: why do you need SFU Server?" We understood the basic principles and necessity of WebRTC SFU server, solved What and Why, and this article went a step further to explore the key technical points for implementing SFU server. Focus on How.

1 what is SFU?

First, let's look at the definition of a SFU server again. What is SFU?

The full name of SFU is: Selective Forwarding Unit, which is a server program that routes and forwards audio and video data streams on WebRTC clients.

As shown in the figure, the core function of the SFU server is to establish a link with each WebRTC Peer client, receive audio and video data from them respectively, and realize the ability of one-to-many (that is, to forward the flow of one client to other WebRTC Peer clients). Then, if we want to implement such a SFU server, what problems need to be solved and dealt with?

We can imagine how Web servers and LVB servers work. Browser / LVB clients usually need the following steps to complete data exchange with the server:

Through DNS parsing, get the IP address of the server; then connect to the server through the "convention" port (such as 80 or 1935)

The client uses a "convention" signaling protocol (such as HTTP,RTMP) to send a request to the server to prepare for data exchange.

The client starts to uplink data to the server, or the server begins to send data to the client, and after that, the connection is closed by signaling.

Static resource-based data (files, web pages), usually the server reads a copy of the data on the disk and gives it to the client that needs it

For non-static real-time data (such as live streaming), the server copies and forwards it to the desired client in "memory"

Similarly, the interaction between the WebRTC client and the SFU server is inseparable from these steps, especially 4can5, which is actually the so-called one-to-many capability.

2. Establishment of signaling and transmission channels

First of all, let's solve the first problem, that is, how does the WebRTC client establish a data transfer channel with the SFU server?

As shown in the figure, let's take a look at the process of establishing a connection between the browser and the Web server: the browser parses the domain name in the URL through DNS, gets the IP and connects to the server through port 80 (this TCP link is reused for subsequent data transmission).

WebRTC is actually similar, but there are some differences compared with the standard HTTP service or RTMP LVB service, as follows:

Signaling and data channels are "separate". At present, there is no unified implementation scheme for signaling, and any scheme (such as HTTP, TCP Custom Protocol, SIP, etc.) can be used, but the data does not go through this signaling link, but through a separate UDP port.

The UDP protocol used in the data channel, unlike the concept of "connection" in TCP, the client only knows the UDP port of the server, but without "connection", it is impossible to predict whether the transport channel is really OK (mainly due to the limitation of some types of NAT gateways, so that not all UDP transmissions can be connected), so we need to use some frameworks and protocols to judge the availability of UDP channels (that is, ICE protocol).

With the above analysis, we can look at how to implement the signaling and transmission channel of SFU:

Implement HTTP Web Server services (either SIP or based on TCP custom protocol) to provide "signaling" support (such as push command, pull command, etc.)

Implement the ICE protocol through the libnice library or your own coding, which is used to provide UDP "data channel" detection and federation.

Implement UDP data monitoring and sending, which is used to receive data from clients and forward data from other clients

3 what "signalling" needs to be realized?

For the HTTP protocol, the "signaling" implemented includes: GET,POST,DELETE, etc., which defines the behavior expected by the browser. Similarly, for SFU, we also need to define a series of necessary signaling to agree on the corresponding behavior of the client and the server.

In fact, what the WebRTC client needs to negotiate with the SFU server is nothing more than the following:

ICE Union: exchange ICE information (username, password, IP address, UDP port, etc.)

Publish stream / unpublish stream: the client informs the server that it is ready to receive data

Subscribe flow / unsubscribe stream: the client informs the server that it is ready to forward data

Therefore, the SFU server can provide ICE Connection/Publish/Subscribe signaling in any way (HTTP/TCP, etc.). The logic behind the signaling of SFU is as follows:

1. ICE Connection: add a UDP channel

2. Publish: add a logical data Producer, recv the client's data through the UDP channel, and notify the logical Consumers

3. Subscribe: add a logical data Consumer. After receiving the Producer notification, send to the client through the UDP channel.

4 how to realize one-to-many?

This is the core feature of SFU, but it is not unique to WebRTC SFU. As mentioned earlier, all non-static resource-based services (real-time data generation and real-time consumption) need to implement one-to-many on the server. A typical example is the RTMP live streaming server, which needs to forward the data pushed by the client to multiple pull clients in real time.

The most important thing to implement one-to-many is to associate the Publisher of data with the Subscriber of data. How to do that?

The audio and video data transmitted by WebRTC is actually encapsulated in the RTP packet. There is a very important field in the RTP header, which is called * × C (synchronous source ID), which is the unique ID of this stream, as shown in the figure:

The producer of data (Publisher) and the consumer of data (Subscriber) can be associated with each other through × × C. the core code logic abstraction of one-to-many is as follows:

That is, when SFU receives the data sent by Publisher, it polls all the Subscribers, and if the match is successful, it forwards the data to the client.

5 data transfer protocol

WebRTC uses the standard RTP/RTCP protocol for data packet and network status feedback. Therefore, the SFU server also needs to support the packet and unpack of RTP/RTCP, so that it can "understand" the meaning of the client's UDP packet, such as extracting necessary information such as * × C or timestamp, and feedback the network status (RTCP) to the client in a timely manner.

RTP/RTCP transport protocol, which has been developed for many years, is a relatively mature multimedia transport protocol, and there are many good open source libraries, so I will not repeat them here.

6 Summary

The above is about how to achieve the core knowledge of the SFU server, for the time being to share here, if you have questions, welcome to write to lujun.hust@gmail.com to communicate. In addition, you are welcome to follow my Sina Weibo @ Lu _ Jun or Wechat official account @ Jhuster for the latest articles and information.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.