Shulou(Shulou.com)06/02 Report--

1.1 what is real-time synchronization: how to achieve real-time synchronization

To use the monitoring service (inotify) to monitor changes in information in the directory of synchronous data servers

If you find a change in the data in the directory, use the rsync service to push it to the backup server.

1.2 the method of realizing real-time synchronization

Data synchronization based on inotify+rsync

Realizing real-time data synchronization by sersync details refer to: the method of realizing real-time data synchronization by sersync

1.2.1 introduction to the principle of real-time synchronization

1.3 data synchronization based on inotify+rsync

1.3.1 introduction to Inotify

Inotify is powerful and fine-grained. Asynchronous file system event monitoring mechanism, linux kernel from 2.6.13, added Inotify support, through Inotify can monitor file system add, delete, modify, move and other events, using this kernel interface, third-party software can monitor various changes of files under the file system, and inotify-tools is the implementation of such monitoring software. Chinese Zhou Yang has also developed a similar real-time synchronization software sersync in Jinshan Company.

Prompt message:

Sersync software is actually developed on the basis of inotify software, and its function is more powerful, including timing retransmission mechanism, filtering mechanism, providing interface to do CDN, and supporting multithreading.

Inotify is actually an event-driven mechanism, which provides a real-time response mechanism for applications to monitor file system events without the need for polling mechanisms such as cron to obtain events. Cron and other mechanisms not only can not achieve real-time, but also consume a lot of system resources. In contrast, based on event-driven, inotify can achieve real-time response to event processing, and there is no consumption of system resources caused by polling. It is a very natural event notification interface and is consistent with the natural world event mechanism.

There are several softwares to implement inotify:

Inotify-tools,sersync,lrsyncd

1.3.2 how inotify+rsync is used

Monitoring of synchronous data Catalog Information by inotify

Rsync completes the real-time synchronization of data information.

Combine with scripts

1.4 prerequisites for deploying inotify software

Kernel versions later than 2.6.13 are required to support inotify software. After kernel 2.6.13, you should have these three files before you install the inotify software.

[root@backup] # ll / proc/sys/fs/inotify/total 0 RW Oct Oct RW max_user_watches-1 root root 0 Oct 17 10:12 max_queued_events-rw-r--r-- 1 root root 0 Oct 17 10:12 max_user_instances-rw-r--r-- 1 root root 0 Oct 17 10:12 max_user_watches

1.4.1 description of three important documents

The effect of file defaults indicates that max_user_watches8192 sets the number of files that the inotifywait or inotifywatch command can monitor (single process) max_user_instances128 sets the number of processes that each user can run inotifywait or inotifywatch commands max_queued_events16384 sets the number of events that the inotify instance event (event) queue can hold

1.4.3 [official statement] three important documents 1.4.2 [Service Optimization] can increase the value of the three files to monitor a wider range

[root@nfs01] # man proc/proc/sys/fs/inotify (since Linux 2.6.13) This directory contains files max_queued_events, max_user_instances, and max_user_watches, that can be used to limit the amount of kernel memory consumed by the inotify interface. For further details, see inotify (7).

Find the detailed description of inotify's default file in level 7 of the man manual.

[root@nfs01] # man 7 inotify/proc/sys/fs/inotify/max_queued_events The value in this file is used when an application calls inotify_init (2) to set an upper limit on the number of events that can be queued to the corresponding inotify instance. Events in excess of this limit are dropped, but an IN_Q_OVERFLOW event is always generated./proc/sys/fs/inotify/max_user_instances This specifies an upper limit on the number of inotify instances that can be created per real user ID./proc/sys/fs/inotify/max_user_watches This specifies an upper limit on the number of watches that can be created per real user ID.

1.5 introduction of inotify software and description of parameters

1.5.1 two installation methods

1) yum install-y inotify-tools

2) compile and install manually

Note:

Epel source is required for YUM installation

Http://mirrors.aliyun.com

Manual compilation and installation requires downloading the software package on github.

Links to Resources for the inotify software:

Https://github.com/rvoicilas/inotify-tools/wiki

1.5.2 two main software installed by inotify

Inotifywait: (main)

Wait for a specific file system event (open close delete, etc.) to occur on the monitored file or directory, and then block it after execution, which is suitable for use in shell scripts.

Inotifywatch:

Collect statistics used by the monitored file system, which refers to the number of file system events that occur.

Description: in real-time real-time synchronization, we mainly use inotifywait to monitor the directory

1.5.3 inotifywait command parameter description

Parameter meaning-m,-- monitor (important parameter) Keep listening for events forever. Without this option, inotifywait will exit after one event is received. Always keep event monitoring. -d,-- daemon111-r,-- recursive (important parameter) Watch all subdirectories of any directories passed as arguments. Recursively monitor directory data information changes-o,-- outfile Print events to rather than stdout. Print the event to the file, equivalent to the standard correct output-s,-- syslogOutput errors to syslog (3) system log module rather than stderr. Sending an error to syslog is equivalent to the standard error output-Q,-- quiet (important parameter) If specified once, the program will be less verbose. Specifically, it will not state when it has completed establishing all inotify watches. Output less information (only print event information)-- exclude Exclude all events on files matching the extended regular expression. Exclude files or directories-- excludei Like-- exclude but case insensitive. When excluding files or directories, it is case-insensitive-- timefmt (important parameter) Print using a specified printf-like format string; read the man page for more details. Specify the time output format-format (important parameter) Print using a specified printf-like formatstring; read the man page for more details. Printing uses the specified output similar to the format string; that is, the actual monitoring output-e (important parameter) Listen for specific event (s). If omitted, all events are listened for. Specifies to listen for the specified event, if omitted The above information can be obtained through inotifywait-- help to obtain 1.5.4-e [parameters] event name event description accessfile or directory contents were read file or directory content is read modifyfile or directory contents were writterv file or directory content is written to attribfile or directory attributes changed file or directory attribute change close_write (important parameter) file or directory closed, after being opened in writeable mode. The file or directory is closed and closed after the write mode is turned on. Close_nowritefile or directory closed, after being opened in read-only mode. File or directory closed, closefile or directory closed, regardless of read/write mode file or directory closed after read-only mode is opened Whether read or write mode openfile or directory opened file or directory is opened moved_to pull file or directory moved to watched directory file or directory is moved to the monitored directory moved_from push file or directory moved from watched directory file or directory is moved from the monitored directory move (important parameter) file or directory moved to or from watched directory file or directory regardless of moving to or out of the monitoring directory triggers the event create (important parameter) file or Directory created within watched directory file or directory created in monitoring directory delete (important parameter) file or directory deleted within watched directory file or directory is deleted in monitoring directory delete_selffile or directory was deleted file or directory is deleted The directory itself is deleted. The information above unmountfile system containing file or directory unmounted can be obtained through inotifywait-- help.

1.5.4.1 event testing in [instance] inotifywait Monitoring

1. Create an event

[root@nfs01 data] # touch test2.txt [root@nfs01 ~] # inotifywait-mrq / data-- timefmt "% d-%m-%y% HV% M"-- format "% T% w% f event Information:% e"-e create17-10-17 11:19 / data/test2.txt event Information: CREATE

2. Delete an event

[root@nfs01 data] #\ rm-f test1.txt [root@nfs01 ~] # inotifywait-mrq / data-- timefmt "% d-%m-%y% HV% M"-- format "% T% w% f event Information:% e"-e delete17-10-17 11:28 / data/test1.txt event Information: DELETE

3. Modify the event

[root@nfs01 data] # echo "132" > test.txt [root@nfs01 ~] # inotifywait-mrq / data-- timefmt "% d-%m-%y% HRV% M"-- format "% T% w% f event Information:% e"-e close_write17-10-17 11:30 / data/test.txt event Information: CLOSE_WRITE,CLOSE

4. Mobile event moved_to

[root@nfs01 data] # mv / etc/hosts. [root@nfs01 ~] # inotifywait-mrq / data-- timefmt "% d-%m-%y% HV% M"-- format "% T% w% f event Information:% e"-e moved_to17-10-17 11:33 / data/hosts event Information: MOVED_TO

5. Mobile event moved_from

[root@nfs01 data] # mv. / hosts / tmp/ [root@nfs01 ~] # inotifywait-mrq / data-- timefmt "% d-%m-%y% HV% M"-- format "% T% w% f event Information:% e"-e moved_from17-10-17 11:34 / data/hosts event Information: MOVED_FROM

1.5.5 inotifywait parameters-parameters defined in format format

The command parameter indicates that when the% w (important parameter) event occurs, the name information of the monitoring file or directory% f (important parameter) event occurs, the file or directory information that triggers the event under the monitoring directory will be displayed, otherwise the event information will be displayed as empty% e (important parameter), different event information will be separated by comma% Xe to display the event information, and different event information will be separated by x. The time format information defined in the output time format of the specified delimiter% T (important parameter) can be modified by X, and the time format specified by-- timefmt option syntax format is matched by the strftime function. The above information can be obtained through inotifywait-- help.

1.5.6 inotifywait parameter-timefmt time format parameter

The command parameter specifies the day of each month for% d (important parameter) The day of the month as a decimal number (range 01 to 31), showing that the double interest is a decimal number (range 01-31)% m (important parameter) The month as a decimal number (range 01 to 12). Display month, display information is decimal (range 01-12)% MThe minute as a decimal number (range 00 to 59). Displays minutes with decimal (range 00-59)% y (important parameter) The year as a decimal number without a century (range 00 to 99). Year information, display information is decimal, and there is no century information% YThe year as a decimal number including the century. Year information, display information is decimal, and contains century information% HThe hour as a decimal number using a 24-hour clock (range 00 to 23). Hour information, display information is decimal, using 24-hour system (range 00-23) description: the above information can be obtained through man strftime information

1.5.6.1 modify the date format of the output

[root@nfs01 ~] # inotifywait-mrq / data-- timefmt "% d/%m/%y% HV% M"-- format "% T% w% f" 11:12 / data/test1.txt on 17-10-17

1.5.7-e [parameters] Summary of important monitoring event parameters:

Important events include event remarks indicating that closeclose_write close_nowrite files or directories are closed, either read or write mode includes write close and read close close_writecreate includes file creation events, but does not contain directory creation events movemoved_to moved_from files or directories trigger events that contain information to move in or out of the monitoring directory important parameters summary: according to the above instructions In practical use, you can create create, delete delete, move movedjto, and modify inotifywait-mrq / data-- format "% w% f"-e create,delete,moved_to,close_write by monitoring the following events.

1.6Tests of inotifywait commands

Description of the inotifywait command test:

Two connection windows need to be opened

1 window runs the inotifywait2 window to operate on the folder, and the monitoring records of inotifywait can be found in one window.

1.6.1. Create the logical ↓ of the file

[root@nfs01 ~] # inotifywait / dataSetting up watches.Watches established./data/ CREATE test1.txt/data/ OPEN test1.txt/data/ ATTRIB test1.txt/data/ CLOSE_WRITE,CLOSE test1.txt creates a file, and inotifywait shows the process of creating a file ↑ [root@nfs01 data] # touch test1.txt

1.6.2 create directory logic ↓

[root@nfs01 data] # mkdir testdir [root@nfs01 ~] # / data/ CREATE,ISDIR testdir

1.6.3 Monitoring the file ↓ in the subdirectory

[root@nfs01 data] # touch testdir/test01.txt [root@nfs01 ~] # inotifywait-mrq / data/ data/testdir/ OPEN test01.txt/data/testdir/ ATTRIB test01.txt/data/testdir/ CLOSE_WRITE,CLOSE test01.txt

1.6.4 sed command modification logic

[root@nfs01 data] # sed's test.txt-I [root@nfs01 ~] # inotifywait-mrq / data-- timefmt "% d-%m-%y% hav% M"-- format "% T% w% f event Information:% e"-e moved_from / data/test.txt event Information: OPEN / data/sedDh6R8v event Information: CREATE / data/sedDh6R8v event Information: OPEN / data/test.txt event Information : ACCESS / data/sedDh6R8v event information: MODIFY / data/sedDh6R8v event information: ATTRIB / data/sedDh6R8v event information: ATTRIB / data/test.txt event information: CLOSE_NOWRITE CLOSE / data/sedDh6R8v event information: CLOSE_WRITE,CLOSE / data/sedDh6R8v event information: MOVED_FROM / data/test.txt event information: MOVED_TO

The sed command replaces logic:

01. Create a temporary file

02. Put the contents of the original file into the temporary file, modify and replace the contents of the temporary file, and leave the original file unchanged.

03. Rename the temporary file to overwrite the original file

1.6.5 Parameter use of-e in inotifywait monitoring

Inotifywait-mrq / data-- timefmt "% d/%m/%y% HVA% M"-- format "% T% w% f event Information:% e"-e create

Description: indicates that only create events are listened for

Inotifywait-mrq / data-- timefmt "% d/%m/%y% HVA% M"-- format "% T% w% f event Information:% e"

Description: do not specify the-e parameter, which means to listen for all events

02. Delete event delete

# inotifywait-mrq / data-- timefmt "% F% H15% M"-- format "% T% w% f event Information:% @ e"-e delete2017-10-17 11:28 / data/02.txt event Information: DELETE2017-10-17 11:28 / data/03.txt event Information: DELETE2017-10-17 11:28 / data/04.txt event Information: DELETE

03. Modify event close_write

# inotifywait-mrq / data-- timefmt "% F% HHD% M"-- format "% T% w% f event Information:% @ e"-e delete Close_write 2017-10-17 11:30 / data/oldgirl.txt event Information: CLOSE_WRITE@CLOSE 2017-10-17 11:30 / data/.oldgirl.txt.swx event Information: CLOSE_WRITE@CLOSE 2017-10-17 11:30 / data/.oldgirl.txt.swx event Information: DELETE 2017-10-17 11:30 / data/.oldgirl.txt.swp event Information: CLOSE_WRITE@CLOSE 2017-10-17 11:30 / data/.oldgirl.txt. Swp event Information: DELETE 2017-10-17 11:30 / data/.oldgirl.txt.swp event Information: CLOSE_WRITE@CLOSE 2017-10-17 11:30 / data/.oldgirl.txt.swp event Information: DELETE

04. Mobile event moved_to

Inotifywait-mrq / data-- timefmt "% F% H15% M"-- format "% T% w% f event Information:% @ e"-e delete,close_write,moved_to2017-10-17 11:34 / data/hosts event Information: MOVED_TO

1.7 schematic diagram of real-time synchronization command parameters

Chapter 2 deployment of inotify+rsync Real-time synchronization Service

2.1 first milestone: deployment of rsync services

2.1.1 rsync server deployment

1) whether the software exists

[root@backup ~] # rpm-qa | grep rsyncrsync-3.0.6-12.el6.x86_64

Requirements: it is very useful to find a command. But I don't know which package it belongs to.

Yum provides rysncprovides Find what package provides the given value

2) configure software services

[root@backup ~] # vim / etc/rsyncd.conf uid = rsyncgid = rsyncuse chroot = nomax connections = 200timeout = 300pid file = / var/run/rsyncd.pidlock file = / var/run/rsync.locklog file = / var/log/rsyncd.logignore errorsread only = falselist = falsehosts allow = 172.16.1.0/24auth users = rsync_backupsecrets file = / etc/ rsync.password [backup] comment = "backup dir by oldboy" path = / backup [nfsbackup] comment = "nfsbackup dir by hzs" path = / nfsbackup

3) create a rsync administrative user

[root@backup] # useradd-s / sbin/nologin-M rsync

4) create a data backup and storage directory, and modify the owner of the directory

[root@backup ~] # mkdir / nfsbackup/ [root@backup ~] # chown-R rsync.rsync / nfsbackup/

5) create an authenticated user password file and authorize 600

Echo "rsync_backup:clsn123" > > / etc/rsync.passwordchmod 600 / etc/rsync.password

6) start the rsync service

Rsync-daemon

At this point, the server configuration is complete

[root@backup] # ps-ef | grep rsync root 2076 1 0 17:05? 00:00:00 rsync-- daemonroot 2163 1817 0 17:38 pts/1 00:00:00 grep-- color=auto rsync

2.1.2 rsync client configuration

1) whether the software exists

[root@backup ~] # rpm-qa | grep rsyncrsync-3.0.6-12.el6.x86_64

2) create a security authentication file and modify the permissions 600

Echo "clsn123" > > / etc/rsync.passwordchmod 600 / etc/rsync.password

3) Test data transmission

[root@nfs01 sersync] # rsync-avz / data rsync_backup@172.16.1.41::nfsbackup-- password-file=/etc/rsync.passwordsending incremental file listdata/data/.hzsdata/.tar.gzdata/.txt

2.2 second milestone: deployment of inotify services

First confirm whether there is an epel source to install inotify-tools software.

[root@nfs01] # yum repolistLoaded plugins: fastestmirror, securityLoading mirror speeds from cached hostfile * base: mirrors.aliyun.com * epel: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.comrepo id repo name statusbase CentOS-6-Base-mirrors.aliyun.com 6706epel Extra Packages for Enterprise Linux 6-x86'64 12401extras CentOS-6-Extras-mirrors.aliyun.com 46updates CentOS-6-Updates-mirrors.aliyun.com 722repolist: 19875

2.2.1 install the inotify software

Two installation methods

1) yum install-y inotify-tools

2) compile and install manually

Note:

Manual compilation and installation requires downloading the software package on github.

Links to Resources for the inotify software:

Https://github.com/rvoicilas/inotify-tools/wiki

2.2.2 View the two commands on the inotify installation (inotifywait,inotifywatch)

[root@nfs01 ~] # rpm-ql inotify-tools/usr/bin/inotifywait # main / usr/bin/inotifywatch

2.2.2.1 the role of inotifywait and inotifywatch:

A total of 2 tools (commands) are installed, namely inotifywait and inotifywatch

Inotifywait: wait for a specific file system event (open close delete, etc.) to occur on the monitored file or directory

It is blocked after execution, which is suitable for use in shell scripts.

Inotifywatch: collect the statistics used by the monitored file system, which refers to the statistics of the number of file system events.

Note: yum can be used directly after installation. If compiling and installing, you need to go to the bin directory of the corresponding software directory to use it.

# Command man manual instructions # man inotifywaitinotifywait-wait for changes to files using inotify uses inotify to monitor, waiting for file information to change # man inotifywatchinotifywatch-gather filesystem access statistics using inotify uses inotify to monitor and collect file system access statistics

2.3 third milestone: writing scripts to achieve the functional combination of rsync+inotify software

2.3.1 rsync service commands:

Rsync-avz-delete / data/ rsync_backup@172.16.1.41::nfsbackup-password-file=/etc/rsync.password

2.3.2 inotify service commands:

Inotifywait-mrq / data-format "% w% f"-e create,delete,move_to,close_write

2.3.3 scripting:

[root@nfs01 sersync] # vim / server scripts az inotify.shroud cards format binapod bashinotifywait-wait / data-- format "% w% f"-e create,delete,moved_to,close_write |\ while read linedo rsync-az-- delete / data/ rsync_backup@172.16.1.41::nfsbackup-- password-file=/etc/rsync.passworddone

Script description:

The for loop defines a condition that stops the loop when the condition is not met

While cycle: continue to cycle as long as the conditions are met

2.3.4 optimize the script

#! / bin/bashPath=/databackup_Server=172.16.1.41/usr/bin/inotifywait-mrq-- format'% w% f'-e create,close_write,delete / data | while read line do if [- f $line]; then rsync-az $line-- delete rsync_backup@$backup_Server::nfsbackup-- password-file=/etc/rsync.password else cd $Path & &\ rsync-az. /-- delete rsync_backup@$backup_Server::nfsbackup-- password-file=/etc/rsync.password fidone

2.4 fourth milestone: test scripting

2.4.1 Let the script run in the background

Create 6 files in the / data directory first

[root@nfs01 data] # sh / server/scripts/inotify.sh & [root@nfs01 data] # touch {1.. 6} .txt

On the backup server, it's time to synchronize six files.

[root@backup] # ll / nfsbackup/total 8txtKuhashi-1 rsync rsync 0 Oct 17 12:06 1.txtKuhashi rwkashi-1 rsync rsync 0 Oct 17 12:06 2.txtKuhashi rkashi-1 rsync rsync 0 Oct 17 12:06 3.txtMashi RWKul-1 rsync rsync 0 Oct 17 12:06 4.txtqrwywashi-1 rsync rsync 0 Oct 17 12:06 5.txt-rw -Rmuri Rafe-1 rsync rsync 0 Oct 17 12:06 6.txt

2.5 script stop method written with while loop statement (kill)

01. Ctrl+z suspends the program, kill-9 kills

02. Do not suspend the program, directly use the killer three swordsmen to kill the process.

Explanation: the three killers of kill are not omnipotent and cannot be killed when the process is paused; kill-9 (dangerous)

2.5.1 check which programs are running in the background

[root@nfs01 data] # jobs [1] + Running sh / server/scripts/inotify.sh &

2.5.2 fg transfers the background program to the foreground

[root@nfs01 data] # fg 1sh / server/scripts/inotify.sh

2.6 the foreground and background running methods of the process:

Fg-- Front desk

Bg-- background

2.6.1 method for running scripts in the background

01. Sh inotify.sh & 02. Nohup sh inotify.sh & 03. Screen implements script programs to run sh / server/scripts/inotify.sh & nohupnohup sh inotify.sh & in the background.

2.7 screen to realize script running in the background

2.7.1 after yum search, it is found that the screen command belongs to the screen package.

[root@test ~] # yum provides screenLoaded plugins: fastestmirror, securityLoading mirror speeds from cached hostfile * base: mirrors.aliyun.com * epel: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.combase | 3.7 kB 00:00 epel | 4.3 kB 00:00 extras | 3.4 kB 00:00 updates | 3.4 kB 00:00 screen-4.0.3-19.el6.x86_64: A screen manager that supports multiple logins on: one terminalRepo: baseMatched from:

2.7.2 install the screen software

[root@test ~] # yum install-y screen

2.7.3 parameters of the screen command

Enter screen in shell to enter the screen view

[root@test ~] # screen

Simple steps for Screen to implement background running programs:

Screen-ls: see the screen session

Screen + a / d: exit the current screen and keep it running in the background

Screen-r ID: specify which screen session to enter

Shortcut keys used in the Screen command

Ctrl+a c: create window

Ctrl+a w: window list

Ctrl+a n: next window

Ctrl+a p: previous window

Ctrl+a 0-9: switch between window 0 and window 9

Ctrl+a K (uppercase): close the current window and switch to the next window

(when exiting the last window, the terminal automatically terminates and returns to the original shell state.)

Exit: close the current window and switch to the next window

(when exiting the last window, the terminal automatically terminates and returns to the original shell state.)

Ctrl+a d: exit the current terminal and return to the status of the shell command before loading screen

Ctrl+a ": window list is different from w

The above inotify+rsync to achieve real-time synchronization method is the editor to share with you all the content, I hope to give you a reference, but also hope that you support more.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.