Shulou(Shulou.com)06/03 Report--

Deployment of rsync+inotify real-time synchronization environment

With the continuous expansion of the scale of the application system, better requirements for the security and reliability of data are also put forward. Rsync has gradually exposed a lot of shortcomings in the high-end business system.

First of all, when rsync synchronizes the data, it needs to scan all the files and compare them for differential transmission. If the number of files reaches the order of millions or even tens of millions, scanning all files will be very time-consuming, and it is often a small number of them that are changing, which is a very inefficient way.

Second, rsync can not monitor and synchronize data in real time. Although it can trigger synchronization through the linux daemon, there must be a time difference between the two triggers, which may lead to inconsistency between server and client data and unable to fully recover data in the event of application failure.

Based on the above two cases, the combination of rsync+inotify can be used to solve the problem, and the real-time synchronization of data can be realized.

Inotify is a powerful, fine-grained, asynchronous file system event control mechanism. Since linux kernel 2.6.13, inotify support has been added. Through inotify, various events such as adding, deleting, modifying, moving and so on can be monitored in the file system. Using this kernel interface, third-party software can monitor various changes of files under the file system, and inotify-tools is the software that implements the monitoring.

After the first full synchronization with rsync, the source directory is monitored in real time with inotify. If there are file changes or new files are generated, they will be synchronized to the target directory immediately, which is very efficient!

Description of requirements:

Separately set

192.168.1.1 / Data/fangfull_upload and / Data/erp_upload192.168.1.2 / Data/xqsj_upload/ and / Data/ fa`n`ghu _ upload_src192.168.1.3 / Data/Static_img/webroot/ssapp-prod and / usr/local/nginx/html/ssapp.prod

Real-time synchronization to

The corresponding fangfull_upload, erp_upload, xqsj_upload, fanghu_upload_src, ssapp-prod and ssapp.prod directories under the / home/backup/image-back directory of 192.168.1.5.

In this way:

(1) the three servers 192.168.1.1, 192.168.1.2 and 192.168.1.3 are source servers and deploy rsync+inotify as clients of rsync.

(2) 192.168.1.5 is the target server and serves as the server of rsync. You only need to install and configure rsync, not inotify.

one。 Deploy the rsync server on the target server 192.168.1.5. 1) disable selinuxvim / etc/selinux/configSELINUX=disabledsetenforce 02) allow the above three source servers on the firewall to access its port 22 and port 873 firewall-cmd-- permanent-- add-rich-rule= "rule family=" ipv4 "source address=" 192.168.1.1 "port protocol=" tcp "port=" 22 "accept" firewall-cmd-permanent-add-rich-rule= "rule family=" ipv4 "source address=" 192.168.1.1 "port protocol=" tcp "port=" 873 "accept" Firewall-cmd-permanent-add-rich-rule= "rule family=" ipv4 "source address=" 192.168.1.2 "port protocol=" tcp "port=" 22 "accept" firewall-cmd-permanent-add-rich-rule= "rule family=" ipv4 "source address=" 192.168.1.2 "port protocol=" tcp "port=" 873 "accept" firewall-cmd-permanent-add-rich-rule= "rule family=" ipv4 "source address=" 192.168.1.3 "port protocol=" tcp "port=" 22 "accept" firewall-cmd-- permanent-- add-rich-rule= "rule family=" ipv4 "source address=" 192.168.1.3 "port protocol=" tcp "port=" 873 "accept" systemctl restart firewalld3) install and deploy rsync service yum install rsync xinetdvim / etc/xinetd.d/rsync.disable = no # from default yes to no Set up boot rsync to start xineted service

/ etc/init.d/xinetd start

Create the / etc/rsyncd.conf file and configure the synchronization policy. Add the following code: vim / etc/rsyncd.conf log file = / var/log/rsyncd.log # log file location, which is automatically generated after starting rsync. There is no need to create the location where the pidfile = / var/run/rsyncd.pid # pid file is stored in advance. Lock file = / var/run/rsync.lock # lock file that supports the max connections parameter secrets file = / etc/rsync.pass # user authentication profile Save the user name and password in it, and then create the file motd file = / etc/rsyncd.Motd # rsync welcome information page file location (create this file yourself, customize the content casually) [fangfull_upload] # Custom name path = / home/backup/image-back/fangfull_upload # rsync server data directory path That is, after synchronizing to the target directory, the storage path comment = fangfull_upload # module name is the same as the [fangfull_upload] custom name uid = nobody # set the uid permission for rsync to run. This should ensure that the permissions synchronized to the target directory are the same as the source directory, that is, they are all nobody! Gid = nobody # sets the gid permissions for rsync to run. Port=873 # default rsync port use chroot = no # defaults to true, modified to no or false Add backup read only for soft connections to directory files = no # set rsync server files to read and write permissions list = no # do not display rsync server resource list max connections = 200 # maximum number of connections timeout = 600 # set timeout auth Users = RSYNC_USER # user name to perform data synchronization Need to be set manually later. You can set multiple, using commas to separate hosts allow = 192.168.1.1 # client IP addresses that allow data synchronization, you can set multiple, use commas to separate hosts deny = 192.168.1.194 # client IP addresses that prohibit data synchronization, you can set multiple addresses, separated by commas in English state (if there is no prohibition There is no need to set this line) [erp_upload] path = / home/backup/image-back/erp_uploadcomment = erp_uploaduid = nobodygid = nobodyport=873use chroot = noread only = nolist = nomax connections = 600auth users = RSYNC_USERhosts allow = 192.168.1.1 [xqsj _ upload] path = / home/backup/image-back/xqsj_uploadcomment = xqsj_uploaduid = nobodygid = nobodyport=873use chroot = noread only = nomax connections = 200timeout = 600auth users = RSYNC_USERhosts allow = 192.168.1.2 [fanghu _ Upload_src] path = / home/backup/image-back/fanghu_upload_srccomment = fanghu_upload_srcuid = nobodygid = nobodyport=873use chroot = noread only = nolist = nomax connections = 600auth users = RSYNC_USERhosts allow = 192.168.1.2 [ssapp-prod] path = / home/backup/image-back/ssapp-prodcomment = ssapp-produid = nginxgid = nginxport=873use chroot = noread only = nomax connections = 200timeout = 600auth users = RSYNC_USERhosts allow = 192.168.1.3 [ssapp.prod] path = / home / backup/image-back/ssapp.prodcomment = ssapp.produid = nginxgid = nginxport=873use chroot = noread only = nolist = nomax connections = 200timeout = 600auth users = RSYNC_USERhosts allow = 192.168.1.3 create user authentication file (multiple modules use the same authentication file) vim / etc/rsync.passxiaoshengyu:123456@rsync set file permissions That is, rsyncd.conf and rsync.pass authentication files are 600 permissions! Chmod / etc/rsyncd.confchmod / etc/rsync.pass restart rsync service / etc/init.d/xinetd restartlsof-i:873COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMExinetd 22041 root 5u IPv6 3336440 0t0 TCP *: rsync (LISTEN) 4) Last Create the destination directory cd / home/backup/image-back/mkdir fangfull_upload erp_upload xqsj_upload fanghu_upload_src ssapp-prod ssapp.prod after rsync synchronization part II: deploy rsync client and inotify monitoring on source servers 192.168.1.1, 192.168.1.2, 192.168.1.3. 1) the same operation of the three machines shuts down selinux, and rsync as the client does not have to open port 873 vim / etc/selinux/configSELINUX=disabledsetenforce 02 in iptables) install rsyncyum install rsync xinetdvim / etc/xinetd.d/rsync.disable = no # by default yes to no in the same operation of the three machines Set the boot rsync to start the rsync service / etc/init.d/xinetd startlsof-i:873COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMExinetd 22041 root 5u IPv6 3336440 0t0 TCP *: rsync (LISTEN) to create a synchronous password file. The file name can be different from the server authentication file, but the password must be the same! Used in rsync synchronization commands. However, it is best to set the files on both sides to be the same, so that it is easy to manage vim / etc/rsync.pass123456@rsync and set the rsync.pass password file to 600 permissions.

Chmod 600 / etc/rsync.pass

3) the three machines operate the same to check whether the server kernel supports inotify, and the following appears Indicates that the server kernel supports inotify ll / proc/sys/fs/inotify max_queued_events max_user_instances max_user_watches. Below, install inotify-toolsyum install make gcc gcc-c++ # install the compiler tool cd / usr/local/srcwget http://github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gztar zxvf inotify-tools-3.14.tar.gzcd inotify-tools-3 .14. / configure-- prefix=/usr/local/inotifymake & & make install sets the system environment variable vim / etc/profile export PATH=$PATH:/usr/local/inotify/bin source / etc/profile to add the library file vim / etc/ld.so.conf/usr/local/inotify/lib ldconfig to modify the inotify default parameter (inotify default kernel parameter value is too small)

View system default parameter values

Sysctl-a | grep max_queued_eventsfs.inotify.max_queued_events = 16384sysctl-a | grep max_user_watchesfs.inotify.max_user_watches = 8192sysctl-a | grep max_user_instancesfs.inotify.max_user_instances = 128.Modification parameter: sysctl-w fs.inotify.max_queued_events= "99999999" sysctl-w fs.inotify.max_user_watches= "99999999" sysctl-w fs.inotify.max_user_instances= "65535" parameter description:

Max_queued_events:

Maximum length of inotify queue. If the value is too small, a "Event Queue Overflow" error will occur, resulting in inaccurate monitoring files.

Max_user_watches:

You can use: find / Data/xqsj_upload-type d | wc-l to count the number of directories under these source directories. You must ensure that the max_user_watches value is greater than the statistical result (where / Data/xqsj_upload is the synchronized source file directory).

Max_user_instances:

Maximum value for each user to create an inotify instance

4) then perform the synchronization operation: perform the first full synchronization of rsync on the three source servers (add-- delete parameter to keep the files in the target directory and the source directory absolutely the same)

On the 192.168.1.1 server

First full synchronization:

Rsync-avH-- port=873-- progress-- delete / Data/fangfull_upload/ RSYNC_USER@192.168.1.5::fangfull_upload-- password-file=/etc/rsync.passrsync-avH-- port=873-- progress-- delete / Data/erp_upload/ RSYNC_USER@192.168.1.5::erp_upload-- password-file=/etc/rsync.pass performs rsync+inotify real-time synchronization script operation after the first full synchronization of rsync+inotify is completed.

The-- delete-before parameter is added to the real-time synchronization script, instead of the-- delete parameter (the parameter used by rsync for the first full synchronization). The difference between the two:

-- delete parameter: indicates that all files in the target directory are forcibly deleted before rsync synchronization, and then the synchronization operation is performed.

-- delete-before parameter: before rsync synchronization, the target directory will be scanned and retrieved, and the redundant files in the destination directory that compare with the source directory will be deleted before performing the synchronization operation. It is obviously safer than the-- delete parameter.

Cd / home/rsync/cat rsyncsides fangfullplates qualified adversaries inotify.shacks. Cd / rsynclockBashSRCDIRcodes backDataapprenticesUSERNCparts USERIPtiles 192.168.1.5DESTDIRFUSERIPBING 192.168.1.5DESTDIRFLOGFULBYBYR d/%m/%y'% H% H%-format'% T% w% f% e'-e close_write,modify,delete,create,attrib Move $SRCDIR | while read filedo/usr/bin/rsync-avH-- port=873-- progress-- delete-before $SRCDIR $USER@$IP::$DESTDIR-- password-file=/etc/rsync.passecho "${file} was rsynced" > > / tmp/rsync.log 2 > & 1donecat rsyncpromoerpacks. :% M'-- format'% T% w% f% e'- e close_write Modify,delete,create,attrib,move $SRCDIR | while read filedo/usr/bin/rsync-avH-- port=873-- progress-- delete-before $SRCDIR $USER@$IP::$DESTDIR-- password-file=/etc/rsync.passecho "${file} was rsynced" > > / tmp/rsync.log 2 > & 1done then start the synchronization script Put it in the background! Nohup sh rsync_fangfull_upload_inotify.sh & nohup sh rsync_erp_upload_inotify.sh & check to see if the script started successfully

Ps-ef | grep inotify

Root 11390 1 0 13:41? 00:00:00 sh rsync_erp_upload_inotify.sh

Root 11392 11390 0 13:41? 00:00:00 sh rsync_erp_upload_inotify.sh

Root 11397 1 0 13:41? 00:00:00 sh rsync_fangfull_upload_inotify.sh

Root 11399 11397 0 13:41? 00:00:00 sh rsync_fangfull_upload_inotify.sh

Root 21842 11702 0 17:22 pts/0 00:00:00 grep-color=auto inotify

Finally, the test is conducted:

For example, creating a file or directory in the source directory / Data/fangfull_upload will automatically synchronize to the target directory / home/backup/image-back/fangfull_upload on the target machine 192.168.1.5 in real time.

On the 192.168.1.2 server

First full synchronization:

Rsync-avH-port=873-progress-delete / Data/xqsj_upload/ RSYNC_USER@192.168.1.5::xqsj_upload-password-file=/etc/rsync.passrsync-avH-port=873-progress-delete / Data/fanghu_upload_src/ RSYNC_USER@192.168.1.5::fanghu_upload_src-password-file=/etc/rsync.pass

Rsync+inotify real-time synchronization:

Cd / home/rsync/cat rsyncsides xqsjacks words admins inotify.shacks packs binptionBashSRCDIRcards packs data banks xqsjdatabases databases adpacks, USERCparts USERIPtiles 192.168.1.5 DESTDIRhearts xqsjcards close_write,modify,delete,create,attrib tags, localinotifywait-mrq-- timefmt'% d/%m/%y% H% H%-format'% T% w% f% e'-e close_write,modify,delete,create,attrib Move $SRCDIR | while read filedo/usr/bin/rsync-avH-- port=873-- progress-- delete-before $SRCDIR $USER@$IP::$DESTDIR-- password-file=/etc/rsync.passecho "${file} was rsynced" > > / tmp/rsync.log 2 > & 1donecat rsynccards fanghuts advertisements admixsrcautomotive inotify.shquestions bindash bins Bash SRCDIRM USERGUBG RSYNCs USERIPPs 192.168.1.5 DESTDIRfanghuipments FanghuFYANGBINOYWER, mrq-timefmt% DUBG M close_write% y% HRV% M'-- format'% T% w% f% e'- e Modify,delete,create,attrib,move $SRCDIR | while read filedo/usr/bin/rsync-avH-- port=873-- progress-- delete-before $SRCDIR $USER@$IP::$DESTDIR-- password-file=/etc/rsync.passecho "${file} was rsynced" > > / tmp/rsync.log 2 > & 1done then start the synchronization script Put it in the background! Nohup sh rsync_xqsj_upload_inotify.sh & nohup rsync_fanghu_upload_src_inotify.sh & finally, test:

For example, creating a file or directory in the source directory / Data/xqsj_upload will automatically synchronize to the target directory / home/backup/image-back/xqsj_upload on the target machine 192.168.1.5 in real time.

On the 192.168.1.3 server

First full synchronization:

Rsync-avH-- port=873-- progress-- delete / Data/Static_img/webroot/ssapp-prod/ RSYNC_USER@192.168.1.5::ssapp-prod-- password-file=/etc/rsync.passrsync-avH-- port=873-- progress-- delete / usr/local/nginx/html/ssapp.prod/ RSYNC_USER@192.168.1.5::ssapp.prod-- password-file=/etc/rsync.passrsync+inotify Real-time synchronization: cd / home/rsync/cat rsync_ssapp-prod _ inotify.shroombinbinfywait-mrq-timefmt'% d/%m/%y% HRV% M'--format'% T% w% f% e'-e Modify,delete,create,attrib Move $SRCDIR | while read filedo/usr/bin/rsync-avH-- port=873-- progress-- delete-before $SRCDIR $USER@$IP::$DESTDIR-- password-file=/etc/rsync.passecho "${file} was rsynced" > > / tmp/rsync.log 2 > & 1donecat rsyncregions ssapp.prodcards inotify.shrooms. & 1donecat rsync _ SRCD _ IRR _ r _ r M close_write% y% HRV% M'-- format'% T% w% f% e'- e Modify,delete,create,attrib,move $SRCDIR | while read filedo/usr/bin/rsync-avH-- port=873-- progress-- delete-before $SRCDIR $USER@$IP::$DESTDIR-- password-file=/etc/rsync.passecho "${file} was rsynced" > > / tmp/rsync.log 2 > & 1done then start the synchronization script Put it in the background! Nohup sh rsync_ssapp-prod_inotify.sh & nohup rsync_ssapp.prod_inotify.sh & finally, test:

For example, creating a file or directory in the source directory / Data/Static_img/webroot/ssapp-prod will automatically synchronize to the target directory / home/backup/image-back/ssapp-prod on the target machine 192.168.1.5 in real time.

If you find an error in the process of synchronization! Repeating the synchronization command always reports this error:

Rsync error: some files/attrs were not transferred (see previous errors) (code 23) at

Main.c (1505)

Finally, the reason was found:

Because when synchronizing, there are soft link files in the source directory!

Rsync synchronizes soft link files, you should add the parameter-l

Therefore, it is best to use the rsync synchronization command followed by a combination of-avpgolr parameters (change the above-avH to-avpgolr)

-a: recursion-v: print detailed process-p: keep file attributes-g: file belongs to group unchanged-o: file belongs to same-l: soft connection properties-r: parameters when synchronizing directories

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.