Shulou(Shulou.com)06/01 Report--

In June, I went to Shanghai to attend the container conference organized by DockOne. I signed up for a small project and accidentally won the first prize. Here, I would like to share with you the construction of the core P2P transmission network, which is also a large file transfer method that the company has put into use.

Project design:

Original intention:

1. As an operator providing global cloud computing business, whether it is the expansion of its overseas node business, platform expansion or customer customized template business, it will involve the transfer of large files. Some common ISO system image files, some customer customized system templates, as little as ten gigabytes, as many as dozens of gigabytes. Such a large file is transferred between multiple nodes around the world, how to ensure its stable and rapid implementation, the most important thing is the requirement of the integrity of the file, which is a problem we objectively face.

2. Traditional file transfer such as FTP and rsync can not meet the requirements of large file transfer. We often encounter a hard day's transfer of a template to a data center node that tells us that the file has been corrupted and cannot be deployed. Reliability cannot be guaranteed.

3. Customers urgently need to achieve fast and reliable distribution and synchronization of large files among nodes around the world.

The idea of realization is:

1. Stable and fast: the stability of the distribution of large files is limited by the transport network and by the transport protocol. Our distribution network is based on the company's dedicated GPN, the stability of the transmission network is guaranteed, and the P2P transmission used in the transport protocol is obviously better than other related protocols such as FTP.

In fact, we can also achieve this through public and private networks.

2. Development: P2P (BT) related software, tools community already has a lot of formed products, so we do not need to develop our own. You only need to choose the appropriate open source tools for secondary development. In the era of cloud computing, development efficiency is the most critical.

3. Deployment delivery: distribution network Docker containerization deployment, image delivery. Based on the lightweight, cross-platform features of Docker, a standard and unified packaging scheme, a deliverable mirror product is generated. Has a good REST API, very suitable for automated testing and continue to integrate, distribute the dynamic expansion of the network.

Project implementation:

1. Network: the document distribution network is built on our company's proprietary GPN global Internet (there is no advertising in the blog, it can actually be achieved in the public network and its own intranet, but pay attention to the speed limit! ).

2. Tracker: centralized P2P transmission mode is adopted, and peertracker is used in tracker. Peertracker is a simple, efficient and fast BT Tracker. Responsible for recording the information of each peer; multiple tracker can be built for redundancy to prevent the cluster from becoming unavailable due to the downtime of one tracker.

3. Peer: uses transmission, is a cross-platform free software with web interface, supports API calls, and has good programmability.

4. Agent: based on flask + dockerapi development, it encapsulates a set of dockerapi-related operations, provides authentication mechanism, controls node scheduling in the receiving center, creates peer docker containers, and performs related tasks. The deployment level is the same as on the transmission peer side.

5. Control node: based on django + celery development. It can monitor the operation of the whole distribution network, the execution of distribution tasks, dynamically expand the distribution nodes, send tasks and other operations.

The whole schematic diagram and flow chart are as follows:

Here we mainly share a playbook, which is used for the construction of P2P network, docker and implementation code, etc., and then shared.

Address of github:

Https://github.com/xujpxm/ansible-peertracker

Due to the haste of writing, with later improvements, the script has been tested to run normally on ubuntu14.

README:

This is a script written by ansible to automate the construction of P2P transmission networks. Tracker uses transmission for the generation and transmission of peertracker,p2p seeds that Google opened up in 2010.

Installation

Install peertracker:

Ansible-playbook install_peertracker.yml

Database created by default: peertracker, user name: peertracker, password: yais6Yab

After the execution of the script, create the database through the web page, as detailed in the installation of peertracker below.

Install transmission:

Ansible-playbook install_transmission.yml

Default rpc_username and password: transmission/transmission

Can modify / etc/transmission-daemon/settings.json file customization.

Create a download directory

Ansible-playbook mk_download_dir.yml

The default download path is: / data/transmission-daemon/downloads can be changed by yourself.

Peertracker

P eerTracker is a simple, efficient and rapid BitTorent Tracker.

Installation requirements:

1. HTTP Web Server.Apache,nginx,lighttpd, etc. as long as php is supported.

2. PHP5+ recommends php5.3 or above

3. Database. The web server that supports MySQL, SQLite3, PostgreSQL8.0, and txtSQL scripts is apache2, and the database uses mysql.

Important Links:

Development Website: http://code.google.com/p/peertracker/

Issue Tracker: http://code.google.com/p/peertracker/issues/list

Source Code Repository: http://peertracker.googlecode.com/svn/trunk/

Github respository: https://github.com/JonnyJD/peertracker

Installation of peertracker

Script an installation environment: ubuntu

1. After confirming that the code is copied to the root of the website, the browser accesses:

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.