Shulou(Shulou.com)06/01 Report--

This article mainly introduces the tutorial of using open source to build mirror websites. In daily operation, I believe that many people have doubts about using open source to build mirror websites. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the doubts of using open source to build mirror websites. Next, please follow the editor to study!

Basic process

Server & bandwidth & Storage

Synchronous source selection, architecture selection

Service provided: ftp | http | rsync 3. Official website to obtain synchronization methods or tools, source list (select upstream source)

Configure synchronization tool

Timing synchronization (crontable)

Backup work, monitoring and alarm, daily maintenance, optimization

Documentation (Wiki), Submit MirrorsList

Other

0/8. Server & bandwidth & Storage

Server: Linux, maybe windows?.. orz

Bandwidth: the larger the better (the uplink bandwidth is too small to provide better service, and the downlink is too small to synchronize completely in time (synchronize in different periods according to network conditions, while avoiding peak load as far as possible). If you want to provide external services, you need a public network IP. It is better to have a domain name.

Storage: the bigger the better (if you provide a large number of services, the home hard drive is not expected to last a year). Consider using SSD as the cache.

1/8. Synchronous source selection, architecture selection

Synchronization source selection:

Choose the distribution you need according to your needs, stability first, diversity second, and you can meet the needs. At the same time, we should also pay attention to the "price-to-performance ratio" (no need to consider self-use images). For example, the performance-to-price ratio such as OpenBSD may be very low, and consuming a large amount of synchronous traffic and disk space may not be very cost-effective. At the same time, for example, mirroring stations in universities need the support of the network center, so you need to consider the support and popularity of users on campus (if the support of students is not enough, the support of nc may also decline). At the same time, there is a certain degree of heat in the school to pass on, so if you add a distribution similar to Ubuntu, "cost-effective" will be very high. Carefully choose the online distribution (estimated by your own resources) and try to keep it as stable as possible. Do not change services at will. Be careful to reserve enough space!

Thank you, brother IGI of CUHK, for your advice. Recommended: "set up linux mirror image"

Architecture selection:

A distribution may protect multiple architectures (for example, debian: alpha, amd64, arm, armel, hppa, hurd-i386, i386, ia64, m68k, mipsel, mips, powerpc, s390, sh and sparc,source,debian existing oldstable,stable, testing,unstable, and experimental versions), so you need to choose the architecture that needs to be synchronized on demand (because disk space is not unlimited), generally choose i386 AMD64, plus source.

2/8. Service provided: ftp | http | rsync

FTP:vsftpd? There are still many, do not quite understand, embarrassed.

HTTP: Webserver recommends nginx or lighttpd, because the image contains a large number of static files, both of which are said to have an advantage over Apache. Remember to change Apache to nginx after cn99 hung up? Suggestion: nginx, lighttpd

Rsync: for downstream users, for example, Tsinghua University of Science and Technology, China University of Science and Technology has opened some distributions, and there are not many mirror stations providing rsync in China. The main reason is that providing rsync will have a high IO load, which is generally unaffordable. You can provide rsync,google if you wish.

3/8. Official website to obtain synchronization methods or tools, source list (select upstream source)

Synchronization methods or tools: usually Rsync, and then pull things from the upstream, you can write your own scripts, or you can use official tools, such as debian's ftpsync, etc. Do not use tools such as wget! Can not guarantee file link and other attributes, can not achieve incremental synchronization, a waste of resources!

Synchronization script of Tsinghua University: http://git.tuna.tsinghua.edu.cn/

The Mirror tool of Beijiao Ideal: https://github.com/ideal/mirror

(thanks to ideal for his help, students who are willing to build an image station can find him on Twitter.) Rubygems,PyPI,npm is also looking for tools or reverse proxies after writing synchronization scripts. In addition to actively pulling, there are official primary images pushed to other official images, such as ArchlinuxARM.

Ubuntu also has a synchronization tool, apt-mirror.

Upstream source:

Get MirrorsList on the official website of the previous release, then look for images that provide rsync services, and select upstream sources, generally from China, Japan, Singapore, Taiwan and other Asia-Pacific regions. Note: the official rsync may only be open to the specified IP (load issues)

4/8. Configure synchronization tool

Write your own synchronization script:

Failed email alarm, synchronization lock control, two-step synchronization, other rsync parameters.

Rsync tutorial: read the synchronization script for igi: http://www.igigo.net/archives/65

Use the synchronization tool:

Synchronization script of Tsinghua University: http://git.tuna.tsinghua.edu.cn/

The Mirror tool of Beijiao Ideal: https://github.com/ideal/mirror/

PyPI,Rubygems,npm:

Read a simple tutorial on Rubygems and PyPI synchronization: http://fangpeishi.com/post/da-jian-pypiji-rubygemsjing-xiang

Read the npm synchronization tutorial: http://maskray.me/blog/2012-11-21-tuna-npm-mirror

Note: to add a synchronization lock, Debian requires two-step synchronization (first synchronize the pool directory).

5/8. Timing synchronization (crontable)

Timing synchronization: throw the written cron script into / etc/cron.d/ "timing automatic Task crontab usage tutorial" Note:

/ etc/cron.d the file names must conform to the filename requirements of run-parts: they must be entirely made up of let-ters, digits and can only contain the special signs underscores (') and hyphens ('-'). Any file that does not conform to theserequirements will not be executed by run-parts. For example, any file containing dots will be ignored. This is done to prevent cron from running any of the files that are left by the Debian package management system when han- dling files in / etc/cron.d/ as configuration files (i.e. Files ending in. DPK g-dist, .dpkg-orig, and. Dpkg-new). (only''and'-'can be used for special characters, plus'.' The first file will be ignored)

Pay attention to the configuration of path in the crontab script, such as using rubygems's synchronization tool to encounter problems such as rake abort.

6/8. Backup work, monitoring and alarm, daily maintenance, optimization

Backup:

When the configuration file changes, use a script to back it up to Dropbox. Monitoring and alarm:

Use awk and other tools to analyze the synchronous diary.

Configure a mail service, such as exim4, to add to the previous script to send mail in the event of an exception. Mobile 139 mailbox has SMS notification function.

Deploy monitoring programs, such as MUNIN, to monitor disk conditions, network conditions, load conditions, etc., and present them visually in the form of charts.

Routine maintenance:

When a synchronization exception fails (for example, upstream down dropped, and upstream network exception, disk exception, etc.), upstream or other methods need to be replaced.

Optimization: mainly for the mirror station for a large number of static files optimization. And observe and monitor the reasonable distribution of traffic and load.

7/8. Documentation (Wiki), Submit MirrorsList

Documentation: one is to accumulate experience, to facilitate inheritance, but also to facilitate other friends who set up mirror stations. For example, build a wiki.

Submit your mirror to mirrorslist:

When the mirror station is stable enough, consider submitting it to the official mirrorslist to let more friends know that it is easy to encourage yourself and inheritance with a small sense of achievement. The images of the intranet can be shared publicly on the internal communication platform.

If the campus mirror station needs to pay attention to publicity, in order to get more support from teachers and students, and facilitate inheritance at the same time.

8/8. Other

If you want to build a mirror station in a university, you need to obtain the support of the network center, which may be required:

A "patient" heart

Pay attention to the inheritance problem (the heat in the school will affect the support of nc to the mirror station, do a good job in documentation and develop good brothers and sisters, if there is a platform organized by Linux Association, it would be best)

Find reliable teacher support (when the mirror is dying, the teacher may be able to save it), be familiar with the teachers or leaders in nc (it is more convenient to solve problems), remember that the mirror station should help promote nc, remember to thank students, teachers and organizations who help

Stability comes first.

At this point, the study on "using open source to build a mirror website tutorial" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.