Shulou(Shulou.com)06/01 Report--

This article mainly shows you the "Linux how to use inotify+rsync to achieve batch file updates", the content is easy to understand, well-organized, hope to help you solve your doubts, the following let the editor lead you to study and learn "how to use inotify+rsync to achieve file batch updates under Linux" this article.

Batch synchronization and batch updates should be the work often done by friends who do operation and maintenance. Rsync is a very powerful synchronization tool under linux. It uses the method of differential synchronization to upload only different parts of files / folders. At the same time, the uploaded part can be compressed first, so the transmission efficiency of rsync is very high, but rsync also has shortcomings. The biggest problem is that it traverses the target directory every time the rsync command is executed. When there are not many files, this is not a problem, once the file count reaches a certain size, then each traversal will consume a lot of resources.

If you only synchronize the frequently changed directories, you can also ignore this problem. If the directories changed each time are larger, then inotify will be used. Inotify is a powerful, fine-grained, asynchronous file system event monitoring mechanism. Since 2.6.13, the Linux kernel has added support for Inotify. Through Inotify, you can monitor all kinds of events in the file system, such as add, delete, modify, move, etc. But inotify only provides C language interface, so it is not convenient to call, so we need to install inotify-tools first.

System environment

CentOS_5.7-x86_64 update source server: 192.168.9.227 destination server: 192.168.9.226 192.168.9.228...

Destination server configuration

192.168.9.226 192.168.9.228 (rsync server):

Check to see if rsync is installed

Rpm-qa | grep rsync

If it is not shipped, install it with the following command

Yum-y install rsync

Define rsync profile / etc/rsyncd.conf

192.168.9.226:

Cat > > / etc/rsyncd.conf log file = / var/log/ rsyncd.log [Web1] path = / data/www1/ignore errorsread only = nolist = nohosts allow = 192.168.9.0/255.255.255.0auth users = www1secrets file = / etc/www1.pwdEOF

192.168.9.228:

Cat > > / etc/rsyncd.conf log file = / var/log/ rsyncd.log [web2] path = / data/www2/ignore errorsread only = nolist = nohosts allow = 192.168.9.0/255.255.255.0auth users = www2secrets file = / etc/www2.pwdEOF

Detailed explanation of rsyncd.conf configuration file

Uid = nobody / / the user running the RSYNC daemon gid = nobody / / the group use chroot running the RSYNC daemon = 0 / / does not use chrootmax connections = 0 / / the maximum number of connections, 0 is not limited to port = 873 / / the default port 873 below these files are automatically generated after the installation of the RSYNC service pid file = / var/run/rsyncd.pid / / pid file storage location lock file = / var/run/rsync.lock / / lock file location. Specify the lock file that supports the max connections parameter. The default value is / var/run/rsyncd.lock.log file = / var/log/rsyncd.log / / the location of the log file Timeout = 300. This option overrides the IP timeout specified by the customer. This option ensures that the rsync server does not wait forever for a crashed client. The timeout unit is seconds, and 0 indicates that there is no timeout definition, which is also the default value. For anonymous rsync servers, an ideal number is 600.Log format =% t% a% m% f% b. Through this option, users can customize the fields of the log file themselves when using transfer logging. Its format is a string that contains format definers. The format definitions that can be used are as follows:% h remote hostname% a remote IP address% l file length characters% p process id%o operation type for this rsync session: "send" or "recv"% f file name% P module name% m module name% t current time% u authenticated user name (null when anonymous)% b actual number of bytes transferred c when the file is sent The default log format for the file recorded in this field is:% o% h [% a]% m (% u)% f% l ". Generally speaking,"% t [% p] "is added to the header of each line. A perl script called rsyncstats is also published in the source code to count log files in this format. # transfer logging = yes causes rsync servers to use files in ftp format to record download and upload operations in their own separate logs. Syslog facility = local3 specifies the message level when rsync sends log messages to syslog. Common message levels are: uth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, sys-log, user, uucp, local0 Local1, local2, local3,local4, local5, local6 and local7. The default value is daemon. Module parameter [web1] / / here is the authenticated module name. On the client side, you need to specify the directory where path = / data/www1/ needs to be mirrored, which is indispensable! Comment = backup web / / comment information for this module ignore errors / can ignore some extraneous IO errors read only = yes / / this option sets whether customers are allowed to upload files. If it is true, then any upload request will fail, if it is false and the server directory read and write permissions are allowed, then upload is allowed. The default value is true.list = no / / does not allow the column file auth users = bak / / authenticated user name, if not this line indicates anonymous, this user has nothing to do with the system this option specifies a list of user names separated by spaces or commas, and only these users are allowed to connect to the module. The users here have nothing to do with the system users. If "auth users" is set, the client's connection request for the module will later be authenticated by the rsync request challenged to authenticate the challenge/response authentication protocol used here. The user's name and password are stored in clear text in the file specified by the secrets file option. By default, modules can be connected without a password (that is, anonymously). Secrets file = / etc/www1.pwd / / password and user name comparison table, the password file generates its own option to specify a file that defines the user name: password pair. This file is useful only if "auth users" is defined. Each line of the file contains a username:passwd pair. Generally speaking, the password should not exceed 8 characters. There is no default secures file name, you need to specify one in a limited format (for example: / etc/www1.pwd). Note: the permissions of this file must be 600, otherwise the client will not be able to connect to the server. Hosts allow = 192.168.9.0 / 255.255.255.0 / allow hosts or network segments this option specifies which IP customers are allowed to connect to the module. The definition of customer mode can be in the following form: a single IP address, for example: 192.168.9.227 entire network segment, for example: 192.168.9.0 IP 24, or 192.168.9.0 IP 255.255.255.0 or network segments need to be separated by spaces, "*" means all, and all hosts are allowed to connect by default. Hosts deny = 0.0.0.0and0 / disable hosts

Establish certification document / etc/www1.pwd

This file must be consistent with the file name specified in the configuration file. The format here is: username:password, security issues, and it is not recommended to use root user 192.168.9.226 in actual use:

Echo "www1:741852" > > / etc/www1.pwd

192.168.9.228:

Echo "www2:951753" > > / etc/www2.pwd

And we need to set the permissions for this file to 600

Chmod 600 / etc/www1.pwdchmod 600 / etc/www2.pwdchmod 600 / etc/rsyncd.conf

Create a motd file (optional)

Rsyncd.motd records welcome messages for rsync services, in which you can enter any text information, such as:

Echo "Welcome to use the rsync services!" > > / var/rsyncd.motd

Start rsync

/ usr/bin/rsync-- daemonecho "/ usr/bin/rsync-- daemon" > / etc/rc.local

Update source server configuration

192.168.9.227 (rsync client)

File system events that inotify can monitor include

IN_ACCESS, that is, files are accessed IN_MODIFY, files are writeIN_ATTRIB, file attributes are modified, such as chmod, chown, touch, IN_CLOSE_WRITE, writable files are closeIN_CLOSE_NOWRITE, unwritable files are closeIN_OPEN, files are openIN_MOVED_FROM, files are removed, such as mvIN_MOVED_TO, files are moved, such as mv, cpIN_CREATE, create new files IN_DELETE, files are deleted, such as rmIN_DELETE_SELF, self-delete That is, an executable file deletes its own IN_MOVE_SELF during execution and moves itself, that is, an executable file moves its own IN_UNMOUNT during execution, the host file system is umountIN_CLOSE, the file is closed, which is equivalent to (IN_CLOSE_WRITE | IN_CLOSE_NOWRITE) IN_MOVE, and the file is moved, which is equivalent to (IN_MOVED_FROM | IN_MOVED_TO).

Note: the files mentioned above also include directories.

Install inotify-tools

Before installing inotify-tools, please make sure that your linux kernel reaches 2.6.13 and the CONFIG_INOTIFY option is turned on during compilation. You can also detect it by using the following command

Ls / proc/sys/fs/inotify

If there are three items of max_queued_events,max_user_instances,max_user_watches, it means support.

Wget http://cloud.github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gztar xvf inotify-tools-3.14.tar.gzcd inotify-tools-3.14./configuremake;make install

Write rsync monitoring script

Vi / rootswap rsync. Shost1192.168.9.226host2q192.168.9.228srcblade www.wt modify,delete,create des1 "web1des2" web2user1 "www1user2" www.w2pxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Attrib\ ${src}\ | while read filedorsync-vzrtopg-- delete-- progress ${src} ${user1} @ ${host1}:: ${des1}-- password-file=/etc/www1.pwd & & rsync-vzrtopg-- delete-progress ${src} ${user2} @ ${host2}:: ${des2}-- password-file=/etc/www2.pwd & & echo "${files} was rsynced" > / tmp/rsync.log 2 > & 1echo "- -"done-m" That is,-- monitor, which means to keep event listening state at all times. -r, that is,-- recursive, represents the recursive query directory. -Q, that is,-- quiet, means to print out monitoring events. -e, namely-- event. This parameter allows you to specify the events to be monitored. Common events include modify, delete, create, attrib, etc.-- timefmt: output format of specified time-- format: specify the details of the change file.

Establish authentication file (only add password for rsync client authentication file)

Echo "741852" > > / etc/www1.pwdecho "951753" > > / etc/www2.pwdchmod / etc/www1.pwdchmod 600 / etc/www2.pwd/bin/sh-n / root/rsync.sh / / Syntax check chmod + x / root/rsync.shnohup sh / root/rsync.sh & echo "nohup sh / root/rsync.sh &" > > / etc/rc.local

Synchronous test

Create a new file on the update source server and run the following command to see if the files can be synchronized properly and to see if there is an error message

Rsync-vzrtopg-- delete-- progress / data/www1/ www1@192.168.9.226::web1-- password-file=/etc/www1.pwd

Submit the updated files to the update source server, so that the updated files will be synchronized to all destination servers in batches through inotify+rsync, which is quite convenient and quick.

The above is all the contents of the article "how to use inotify+rsync to update files in batch under Linux". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.