Shulou(Shulou.com)06/01 Report--

This article mainly explains "what are the main technologies used in Twitch". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Now let the editor take you to learn "what are the main technologies used in Twitch?"

Twitch is a real-time streaming video platform for video games, co-founded by Justin Kan and Emmett Shear. It is an independent operating site of Justin.tv that focuses on game-related content. According to its in-house analysts, Twitch receives more than 38 million visitors a month, more than 20 million gamers converge on the platform, and each visitor stays on the site for an average of 1.5 hours a day. The website supports the languages of 28 countries and regions, including simplified and traditional Chinese.

Twitch's live broadcast mode is completely different from YouTube and other batch-on-demand processing methods, and live broadcasting requires higher and more difficult technology. this is also the reason why domestic TV live streaming still depends on cable network, while the live TV broadcast business on the Internet is greatly reduced in terms of live broadcast effect. Twitch has explored a successful way to use Internet technology to achieve smooth and uninterrupted live broadcasting.

Twitch LVB video is different from YouTube batch video: the latter stores all videos on disk and replays them later according to requirements, while LVB video storage, writing and video reading and playback are carried out at the same time, so a completely different architecture is required. Here is the technology stack:

Usher-this is its core system, which is used to implement the business logic server for video stream playback.

Twice-customizable web caching system (http://code.google.com/p/twicecache/))

XFS-the file system stores video in seconds in the system

HAProxy-Software load balancing.

LVS stack and ldirectord-ensure high availability.

Ruby on Rails-Application Server

Nginx-web server

PostgreSQL-stores users and other metadata

MongoDB-used to store user action events for internal analysis

MemcachedDB-used to handle highly intensive write operations such as browsing

Syslog-ng-Log Service

RabitMQ-for job systems.

Puppet-used to build the server.

Git-source control.

Wowza-Flash/H.264 video server, many custom modules are written in Java

S3-small image storage.

Following in the footsteps of YouTube and other manufacturers, even the game live streaming service Twitch has "begun" to abandon Flash and use HTML5. According to the official website, Twitch has completed the first step, first changing the old Flash module into a combination of HTML5 and Javascript, and redesigning the playback control interface. Since this is the first step, it means that the Twitch video itself is still based on Flash, so the next step is to gradually replace the player with the HTML5 foundation from the inside to the outside. The new interface is already available on the Channel page and is gradually being pushed to users, so don't think you're going to the wrong site when you see the interface getting different.

One question is: why is live video so difficult? It seems that it only needs a lot of bandwidth, so it's all right to combine the video around the stream in memory, but it's not that simple. What makes live video so challenging?

1. Video cannot be interrupted like hiccups, and if the video exceeds the network capacity for even a fraction of a second, every viewer will see "buffering" on the screen at the same time. It is very important to have network capacity.

two。 Overflow needs CDN implementation overflow Usher will handle this logic, once the number of users exceeds the maximum capacity, the new player will be sent to the CDN server.

3. When the audience quickly finds any problems, they will talk and chat immediately. Users expect to be able to deal with these problems gracefully. They have to wait until everyone on a server has finished browsing before they can let the server maintain the mode. This is a very slow maintenance process. The session must be uninterrupted. A typical website can have many errors that few people will notice, but the live broadcast system is different.

Let's take a look at how Twitch meets these challenges.

Their biggest problem is to control the flash crowd, the so-called flash crowd, when many people want to see the same thing at the same time. This is a huge amount of incoming traffic. Therefore, they need to create a method to achieve real-time adaptive load between all video servers and data centers. The mechanism is Usher.

Usher is a piece of software they developed to manage other business logic such as load balancing authorization and playback. For each streaming video, Usher calculates how many servers are sending them to ensure the best load. It determines in real time how to replicate streams between these servers, based on the following rules:

Separate load on all servers

Optimized delay

Which servers does a stream flow on?

The user's IP address, so that you can tell which country the user is from

According to the routing route database, find the nearest ISP to the user IP.

Try to send the request to a video server in the same data center, depending on the data center from which the request came.

Use these optimization metrics to guide the optimization of each request to the server to ensure better latency and performance optimization. They also have a lot of monitoring dials and very fine-grained controls.

Each server can act as an edge server (where video is sent directly to viewers) and a source server (video streams from a broadcast to the server). Continuous dynamic adjustments are made based on the load policy that a flow can apply to one server or each server in the network.

The connection of replication streams between servers is like a tree structure, and the number of streams is constantly sampled. If the new browsing of a stream increases rapidly, the flow will be copied to other servers, and the process will be repeated to build a tree (banq Note: according to the construction law, the tree is the most effective life system feature), which may eventually cover all servers in a network, and this process is performed every three seconds.

The entire video stream is in memory all the time from its source server to the copy to another server and to the user without any disk storage.

Using RTMP protocol (video stream playback protocol), each stream needs an independent session, which will bring expensive overhead, but broadcast multicast and P2P technology are not used. Many downstream ISP do not support multicast, but only use multicast to copy video on the internal server. The internal bandwidth is very cheap, but it does not have many benefits, because it can not be fine-grained to control replication between servers.

According to the HTTP request, Usher decides which server will handle the requested video, while the video server is generally passive, and Usher controls the topology of the whole server before it.

The video stream does not come from the disk. The video is archived and stored on the disk. The source server is selected to process a new video stream uploaded, recording the stream on the local disk, and the video is saved and archived every second. The archive storage server uses the XFS file system. The architecture can handle thousands of concurrent video incoming writes. Each video stream is saved by default for 7 days, and video files may be saved across disk partitions.

Migrating from other weight protocols to HTTP streaming protocols is happy and can be well extended using existing technologies, but there is a problem that must be actively faced, that is, delay and real-time problems, usually people think that no more than 5-30 seconds is real-time, but this does not apply to thousands of people real-time communication interaction, can not have a delay of 1jump 4 seconds.

The above is an introduction to the video broadcast replication system, and they also have a set of Web architecture. The two architectures are as follows:

At this point, I believe you have a deeper understanding of "what are the main technologies used in Twitch?" you might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.