Shulou(Shulou.com)06/01 Report--

This article shows you how to understand the Sersync server synchronization program project and design framework, the content is concise and easy to understand, it will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

This project uses inotify and rsync to synchronize the server in real time, in which inotify is used to monitor file system events. Rsync is a widely used synchronization algorithm, and its advantage is that it only operates on different parts of the file, so its advantage is much more than the use of hanging file system for image synchronization.

At present, the more commonly used version of the synchronous program is inotify-tools, and the other is the google open source project Openduckbill (which depends on inotify-tools), both of which are written based on scripting languages, and their design ideas are also using inotify and rsync commands. Compared with the above two projects, the advantages of this project are:

1.sersync is written in C++ and filters temporary files and repeated file operations generated by the linux system file system (I'll talk about it later), so when combined with rsync synchronization, it saves runtime and network resources. So it's faster.

two。 Compared to the above two projects, sersync configuration is very simple: download the source code at http://code.google.com/p/sersync/downloads/list (divided into 32 versions and 64-bit versions), in which the bin directory already has my compiled binary files, which can be used directly with the xml files under the bin directory.

3. In addition, compared with other scripted open source projects, this project uses multi-threads to synchronize, especially when synchronizing larger files, it can ensure that multiple servers keep synchronized in real time.

4. This project has its own error handling mechanism, through the failure queue to re-error the wrong files, if it still fails, then resynchronize the failed files every 10 hours.

5. This project has its own crontab function, and as long as it is enabled in the xml configuration file, it can be synchronized as a whole at regular intervals according to your requirements.

6. This project comes with socket and http protocol extensions to meet your secondary development needs.

Basic architecture:

A brief Analysis of Design

As shown in the figure above, the thread group thread is the daemon thread waiting for the thread queue. When there is data in the queue, the thread group daemon thread wakes up one by one, and when there are many inotify events in the queue, it will be awakened to work together. The purpose of this design is to be able to handle multiple inotify events at the same time, resending to take advantage of the concurrency ability of the server (core number * 2x2).

The reason why it is called thread group thread is that when each thread is working, it will establish child threads according to the number of servers, and child threads can ensure that all files are synchronized with each server at the same time. When the files to be synchronized are large, this design ensures that each remote server can get the files to be synchronized at the same time.

The service thread has three functions, the first is to deal with the files that failed to synchronize, synchronize these files again, and generate a rsync_fail_log.sh script to record the failed events for the files that fail to synchronize again. At the same time, the script is executed every 10 hours, while emptying the script. The third function of service threads is the crontab function, which synchronizes all paths as a whole at regular intervals.

The establishment of the filter queue is to filter the duplicate inotify information generated in a short period of time. For example, when deleting a folder, inotify will generate both delete folder and delete folder events. By filtering the queue when the delete folder event occurs, it will filter out all the previously queued events of deleting files, so that only one event will be generated to reduce the burden of synchronization. At the same time, temporary files and repeated operations will be generated when modifying files.

For example:

When we perform a wq operation in a test file in vi, the following event occurs:

Even if you put "." The world at the beginning and the end of "~" is filtered, and there are still three operations for test files, namely delete, create and save. By filtering the queue, there is only one event left, which improves efficiency to a certain extent.

The second function of the filter queue, that is, when you delete a directory locally, suppose you delete a directory with five files, inotify will produce six events, namely, five file deletion events, and one delete directory event. If you use the filter queue, there will normally be only one event to delete the directory, which greatly reduces the number of rsync communications. (of course, this is not absolute. If these six events are read into the queue several times, the synchronization may have been removed from the queue by the synchronization thread before it can be filtered. However, to some extent, it can reduce the number of synchronous communications when deleting folders.

The third function of the filter queue is to filter the folders in the monitoring directory, if you do not want to synchronize some folders in the directory, or some files with suffixes. For the subfolders that do not need to be monitored, remove the monitoring when inotify starts, and for the subfiles that do not need to be monitored, the generated file events will be filtered out before entering the synchronization queue. If you use rsync with-- exclude, you can filter it, but you still interact with the rsync daemon.

The above content is how to understand the Sersync server synchronization program project and design framework, have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.