How to realize Real-time data synchronization by sersync

Editor to share with you how sersync to achieve real-time data synchronization, I believe that most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to know it!

1.1 first milestone: installing sersync software

1.1.1 upload the software to the server and extract it

1. Upload the software to the server rz-E

To facilitate management, the upload location is uniformly set to / server/tools.

2. Decompress the package

[root@backup sersync_installdir_64bit] # tree. └── sersync ├── bin │ └── sersync ├── conf │ └── confxml.xml └── logs

1.1.2 binary package installation method

Binary package installation software method (green software installation method):

You can use it by decompressing it directly.

[root@nfs01 sersync_installdir_64bit] # mv sersync/ / usr/local/ [root@nfs01 tools] # tree / usr/local/sersync/ / usr/local/sersync/ ├── bin │ └── sersync ├── conf │ └── confxml.xml └── logsdirectories, 2 files

1.2 second milestone: writing a sersync configuration file

1.2.1 Common grammatical formats

Rsync configuration file authoring: ini syntax

Sersync configuration file authoring: xml syntax

Ansible configuration file authoring: yml syntax

1.2.2 modify the configuration file

Pre-writing backup

[root@backup conf] # lltotal 4Murray RWMurray Rafael-1 root root 2214 Oct 26 2011 confxml.xml [root@backup conf] # cp confxml.xml {, .bak}

Lines 6-11 indicate excluding synchronized data, which is equivalent to-- exclude function, indicating exclusion.

Line 12-21 uses the function of inotify to monitor specified events, which is equivalent to-e create,delete. Indicates the specified monitoring event information

Line 24-28: where to push the name= module is the address of the rsync server

# Monitoring that directory

Lines 29-35 define the parameter information for rsync push.

Note: do not have word misspelling (true), otherwise the program will not start properly or get stuck.

Final content of the configuration file:

[root@nfs01 tools] # cat / usr/local/sersync/conf/confxml.xml

1.3 third milestone: start sersync

1.3.1 permissions to modify files (executable)

First of all, let the program give the file execution permission.

[root@nfs01 bin] # chmod axix sersync [root@nfs01 bin] # lltotal 1768-rwxr-xr-x 1 root root 1810128 Oct 26 2011 sersync

1.3.2 View help information for the software

[root@nfs01 bin] #. / sersync-hset the system paramexecute:echo 50000000 > / proc/sys/fs/inotify/max_user_watchesexecute:echo 327679 > / proc/sys/fs/inotify/max_queued_eventsparse the command param___ important parameter-d: enable gatekeeper Important parameter of daemon mode-r: before monitoring Push the monitoring directory and the remote host once with the rsync command parameter-n: specify the number of daemon threads on, default to 10 important parameters-o: specify configuration file, default use confxml.xml file parameter-m: enable other modules separately, use-m refreshCDN to enable refresh CDN module parameter-m: enable other modules separately Use-m socket to enable the socket module parameter-m: enable other modules separately, and use-m http to enable the http module without the-m parameter, then the synchronization program _ will be executed by default.

1.3.3 start the program in the bin directory of the program

. / sersync-dro / usr/local/sersync/conf/confxml.xml

1.3.4 Startup method II

Add the bin directory of the / usr/local/sersync/bin/ program to PATH

Export PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/sersync/bin/

Then the sersync command can be used directly

[root@nfs01 scripts] # sersync-dro / usr/local/sersync/conf/confxml.xml set the system paramexecute:echo 50000000 > / proc/sys/fs/inotify/max_user_watchesexecute:echo 327679 > / proc/sys/fs/inotify/max_queued_eventsparse the command paramoption:-d run as a daemonoption:-r rsync all the local files to the remote servers before the sersync workoption:-o config xml name: / usr/local/sersync/conf/confxml.xmldaemon thread num: 10parse xml config filehost ip: localhost host port: 8008daemon start Sersync run behind the console use rsync password-file: user is rsync_backuppasswordfile is / etc/rsync.passwordconfig xml parse successplease set / etc/rsyncd.conf max connections=0 Manuallysersync working thread 12 = 1 (primary thread) + 1 (fail retry thread) + 10 (daemon sub threads) Max threads numbers is: 22 = 12 (Thread pool nums) + 10 (Sub threads) please according your cpu Use-n param to adjust the cpu rate--rsync the directory recursivly to the remote servers onceworking please wait...execute command: cd / data & & rsync-az-R-- delete. / rsync_backup@172.16.1.41::nfsbackup-- password-file=/etc/rsync.password > / dev/null 2 > & 1 run the sersync: watch path is: / data

1.4 Summary and comparison of Inotify and sersync

1.4.1 Inotify real-time concurrency:

Conclusion: after testing, 200 files per second are concurrent, and there is almost no delay in data synchronization (less than 1 second).

1.4.2 advantages of inotify:

1) monitor the changes of file system events and realize real-time data synchronization through synchronization tools.

1.4.3 shortcomings of inotify

1) if the concurrency is greater than 200 files (10-100k), synchronization will be delayed

2) the script we wrote earlier is pushed all once at a time, but it is incremental. You can also only synchronize the changed files and ignore the changes.

3) after monitoring the event, calling rsync synchronization is single-process, while sersync is multi-process synchronization. Why develop sersync when you have inotify-tools?

1.4.4 serysync has many functions: (inotify+rsync command)

1) support management through configuration files

2) the real daemon socket

3) the failed file can be retransmitted regularly (scheduled task function)

4) third-party HTTP interface (for example, update cdn cache)

5) default multi-process rsync synchronization

1.4.5 Summary of real-time synchronization of highly concurrent data:

1) inotify (sersync) + rsync, which is file-level.

2) drbd file system level, file system level, block-based block synchronization, disadvantages: backup node data is not available

3) synchronization function of third-party software: mysql synchronization (master-slave replication), oracle,mongodb

4) the program is double-written, directly writing two servers.

5) solve the problem by using product business logic (separation of read and write, backup can not read, reader)

2.1 level of the man command

Centos6

[root@nfs01 ~] # man man The standard sections of the manual include: User Commands # user command System Calls # system calls C Library Functions # library function Devices and Special Files # device and special file File Formats and Conventions # file format and convention Games et. Al. # games, etc. Miscellanea # Miscellaneous Notes System Administration tools and Daemons # system management tools and programs Distributions customize the manual section to their specifics, which often include additional sections.

Centos7

[root@clsn tuichu] # man ~ Executable programs or shell commands System calls (functions provided by the kernel) Library calls (functions within program libraries) Special files (usually found in / dev) File formats and conventions eg / etc/passwd Games Miscellaneous (including macro packages and conventions), e.g. Man (7), groff (7) System administration commands (usually only for root) Kernel routines [Non standard] these are all the contents of the article "how sersync achieves real-time data synchronization". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!