Shulou(Shulou.com)06/01 Report--

How to understand Linux remote data synchronization tool, in view of this problem, this article introduces the corresponding analysis and answer in detail, hoping to help more partners who want to solve this problem to find a more simple and easy way.

I. brief introduction

1 understanding

Rsync (remote synchronize) is a remote data synchronization tool that allows you to quickly synchronize files between multiple hosts through LAN/WAN. Rsync uses the so-called "Rsync algorithm" to synchronize files between local and remote hosts. This algorithm transfers only different parts of the two files, rather than the whole file every time, so it is quite fast.

Rsync supports most Unix-like systems and has been well tested on Linux, Solaris and BSD.

In addition, it also has corresponding versions under the windows platform, such as cwRsync and Sync2NAS tools.

2 principle

Rsync was originally used as a tool to replace rcp, but it is currently maintained by rsync.samba.org, so the format of the rsync.conf file is similar to the main configuration file of samba.

Rsync can be used through rsh or ssh, or it can be run in daemon mode. When running in daemon mode, Rsync server opens a port 873 and waits for the client to connect.

When connecting, Rsync server checks whether the password matches, and if checked by the password, you can start the file transfer. When the connection is completed, the entire file will be transferred once, and then only an incremental backup will be required.

3 characteristics

You can mirror and save the entire directory tree and file system

It is easy to maintain the permissions, time, soft and hard links of the original file, etc.

Can be installed without special permission

Optimized process, high efficiency of file transfer

You can use rsh, ssh, etc., to transfer files, of course, through a direct socket connection

Support for anonymous transmission

II. Ssh mode

1. Synchronization between locals

Environment: 172.16.22.11 # mkdir src # touch src/ {arecrine bjorcrecy d} # mkdir dest # rsync-av 172.16.22.12:/data/test/src/ dest/-remote synchronization to the local Enter the root password # rsync-av src/ 172.16.22.12:/data/test/dest/-- synchronize local files to the remote # rsync-av src 172.16.22.12:/data/test/dest/-- synchronize the entire directory over # rm-rf src/d-- delete a file d # rsync-av-- delete src/ 172.16.22.12:/data/test/dest/-- delete, and delete extraneous files from the target directory.

2. Synchronization between Lans

Environment: 172.16.22.11

# useradd george # passwd george # mkdir / home/george/test # touch / home/george/test/g {1,2,3,4}

3. Local area network designated user synchronization

-172.16.22.12

# useradd george # passwd george # mkdir / home/george/test # touch / home/george/test/g {1,2,3,4}

-172.16.22.11

# rsync-av src'- e ssh-l george' 172.16.22.12:/home/george-- Local sync to remote # rsync-av 172.16.22.12 HomeGetGetWork *'- e ssh-l george-p 22' dest/

Third, daemon mode

Environment: 192.168.22.11

1. Service startup mode

1.1.The rsync server with heavy load should be run independently.

# yum install rsync xinetd-Service installation # / usr/bin/rsync-daemon

1.2.For lighter-loaded rsync servers, xinetd can be used.

# yum install rsync xinetd-- Service installation # vim / etc/xinetd.d/rsync-- configure the managed service, change the following item to no disable = no # / etc/init.d/xinetd start-- start the managed service xinetd # chkconfig rsync on # netstat-ntpl | grep 873-check whether the service is started

2. Detailed explanation of configuration

Both rsync service operation modes require configuration of rsyncd.conf, which is similar to the main configuration file of samba

Global parameter

A module parameter can also be defined in the global parameters section, and the value of this parameter is the default value of all modules.

Address-the IP address used for the specified server to run when running independently; this parameter is ignored by the xinetd runtime and replaced with the-address option on the command line. Default all local IP

Port-specifies the port number on which the rsync daemon listens. This parameter is ignored by the xinetd runtime and replaced with the-port option on the command line. Default 873

Motd file-specifies a message file whose contents are displayed to the customer when the customer connects to the server

The daemon of pid file-rsync writes its PID to the specified file

Log file-specifies the log file for the rsync daemon without sending the log to syslog

Syslog facility-specifies the message level when rsync sends log messages to syslog

Socket options-specify custom TCP options

Lockfile-specifies the lock file storage path for rsync

Timeout = 600-timeout

Module parameters

Module parameters are mainly used to define which directory of the rsync server will be synchronized. The module declaration must be in the form of [module], which is the name you see on the rsync client, similar to the share name provided by the Samba server. The data that the server actually synchronizes is specified through path.

Basic module parameters

Path-specifies the synchronization path of the current module on the rsync server, this parameter must be specified

Comment-assign a description to the module, which, together with the module name, is displayed to the customer when the customer connects to get the module list

Module control parameters

Use chroot =-defaults to true. Before transferring files, first chroot to the directory specified by the path parameter; advantages, security; disadvantages, root permission is required, and the directory files pointed to by symbolic links outside path cannot be backed up

Uid =-specifies the module to transfer files to the specified UID; default nobody

Gid =-specifies the module to transfer files to the specified GID; default nobody

Number of max connections-* concurrent connections. 0 means no limit.

Lock file-specifies the lock file that supports the max connections parameter. Default / var/run/rsyncd.lock

List-specifies whether the module should be listed when the customer requests a list of modules that can be used. Default is true, display

Read only =-read-only option, that is, clients are not allowed to upload files to the server. Default true

Write only =-write only selections, that is, clients are not allowed to download files from the server. Default false

Ignore errors-ignore IO errors. Default true

Ignore nonreadable-specifies that the rysnc server completely ignores files that the user does not have access to. This makes sense when there are some files in the directory that need to be backed up that should not be obtained by the backup. False

Timeout =-this option overrides the IP timeout specified by the customer. This ensures that the rsync server does not wait forever for a crashed client. For anonymous rsync servers, the ideal number is 600 (in seconds). 0 (unrestricted)

Dont compress-used to specify files that are not compressed before transfer. This option defines a list of command options that customers are not allowed to use on the module. You must use the full name of the option, not the abbreviation. When an option is rejected, the server reports an error message and exits. For example, to prevent the use of compression, it should be: "dont compress =". .gz.tgz .zip.z .rpm.deb .iso.bz2 .tbz

Module file filter parameters

Exclude-specifies multiple files or directories (relative paths) separated by spaces and adds them to the exclude list. This is equivalent to using-exclude in a client command to specify the mode

Exclude from-specifies a file name that contains the exclude rule definition from which the server reads the exclude list definition

Include-specifies multiple files or directories (relative paths) separated by spaces and adds them to the include list. This is equivalent to using-include in a client command to specify the mode

Include from-specifies a file name that contains the include rule definition from which the server reads the include list definition

Module user authentication parameters

Auth users-specifies a list of user names separated by spaces or commas, and only these users are allowed to connect to the module (without any relationship with the system user). The user name and password are stored in clear text in the file specified by the secrets file parameter. The default is anonymous.

Secrets file-specifies an rsync authentication password file. This file works only if auth users is defined. File permissions must be 600

Strict modes-specifies whether to monitor the permissions of the password file. For true, the password file can only be accessed by users who are running on the rsync server, and no other user can access the file. Default is true

Module access control parameters

Hosts allow-use a list of hosts to specify which host customers are allowed to connect to the module. Hosts that do not match the list of hosts will be rejected. The default value is *

Hosts deny-use a list of hosts to specify which host customers are not allowed to connect to the module

Module log parameters

Transfer logging-causes the rsync server to log the transfer operation to the transfer log file. The default value is false

Log format-specifies the field in which the log file is transferred. The default is: "o% h [% a]% m (% u) f% l"

When the "log file" parameter is set, "% t [% p]" is added at the beginning of each line of the log

The log format definers that can be used are as follows:

% o-Action type: "send" or "recv"

% h-remote hostname

% a-remote IP address

% m-module name

User name of% u-certificate (when anonymous)

% f-File name

% l-number of file length characters

% p-PID of this rsync session

% P-Module path

% t-current time

% b-actual number of bytes transferred

% c-record the check code of the file when it is sent

3. Server configuration

# vim / etc/rsyncd.conf-- Edit the configuration file for the rsyncd service. It is not available by default. You need to edit it yourself.

Uid = root-rsync running permission is root

Gid = root-rsync running permission is root

Use chroot = no-whether to let the process leave the working directory

Max connections = 5-* number of concurrent connections. 0 means no limit.

Timeout = 600-timeout

Pid file = / var/run/rsyncd.pid-specifies the pid storage path of the rsync

Lockfile = / var/run/rsyncd.lock-specifies the lock file storage path for rsync

Log file = / var/log/rsyncd.log-specify the log storage path for rsync

[web1]-Module name

Path = / data/test/src-the base path where the module stores the files

Ignore errors = yes-ignores some extraneous Icano errors

Read only = no-client can upload

Write only = no-client can download

Hosts allow = 192.168.22.12-ip of client hosts allowed to connect

Hosts deny =-blacklist, indicating any host

List = yes

Auth users = web-authenticate the user name of this module

Secrets file = / etc/web.passwd-specifies the file to store the "username: password" format

# mkdir / data/test/src-- create the base directory # mkdir / data/test/src/george-- create another directory # touch / data/test/src/ {1meme2} # echo "web:123" > / etc/web.passwd-- create the password file # chmod / etc/web.passwd # service xinetd restart

IV. Testing

1. Client

Environment: 192.168.22.12 # yum-y install rsync # mkdir / data/test

2. Small-scale test parameters

# rsync-avzP web@192.168.22.11::web1 / data/test/-enter password 123; synchronize the files in the server web1 module to / data/test. Parameter description:

-a-parameter, which is equivalent to-rlptgoD

-r.-it's recursive.

-l-is a linked file, which means to copy a linked file

-I-lists the files in the rsync server

-p-means to maintain the original permissions of the file

-t-keep the original time of the file

-g-keep the file original user group

-o-keep the original owner of the document

-D-equivalent to block device files

-z-Compression on transmission

-P-Transmission Progress

-v-progress and other information during transmission, which has something to do with-P

# rsync-avzP-- delete web@192.168.22.11::web1 / data/test/-- keep the client and server exactly the same -- delete # rsync-avzP-- delete / data/test/ web@192.168.22.11::web1-- upload client files to the server # rsync-avzP-- delete / data/test/ web@192.168.22.11::web1/george-- upload client files to the server george directory # rsync-ir-- password-file=/tmp/rsync.password web@192.168.22.11::web1-- recursively list the server File of the server web1 module # rsync-avzP-- exclude= "* 3*"-- password-file=/tmp/rsync.password web@192.168.22.11::web1 / data/test/-- synchronize all files except the path and the file name containing "3" *

3. Synchronize through password files

# echo "123" > / tmp/rsync.password # chmod 600 / tmp/rsync.password # rsync-avzP-- delete-- password-file=/tmp/rsync.password web@192.168.22.11::web1 / data/test/-- call password file

4. Client automatic synchronization

# crontab-e 100 * rsync-avzP-delete-password-file=/tmp/rsync.password web@192.168.22.11::web1 / data/test/ # crontab-l

5. Real-time data synchronization

Environment: Rsync + Inotify-tools

1 、 inotify-tools

Is a set of c development interface library functions for inotify file monitoring tools under linux, as well as a series of command line tools that can be used to monitor file system events

Inotify-tools is written in c and does not depend on other than requiring the kernel to support inotify

Inotify-tools provides two tools: one is inotifywait, which is used to monitor changes in files or directories, and the other is inotifywatch, which is used to count the number of file system visits

2. Install inotify-tools

# yum install-y gcc-installation dependency # mkdir / usr/local/inotify # tar-xf inotify-tools-3.14.tar.gz # cd inotify-tools-3.14 #. / configure-- prefix=/usr/local/inotify/ # make & & make install

3. Set environment variables

# vim / root/.bash_profileexport PATH=/usr/local/inotify/bin/:$PATH # source / root/.bash_profile # echo'/ usr/local/inotify/lib' > > / etc/ld.so.conf-load the library file # ldconfig # ln-s / usr/local/inotify/include / usr/include/inotify

4. Common parameters

-m-always keep listening, and the default triggering event exits

-r-Recursive query directory

-Q-print out monitoring events

-e-defines the monitored events. Available parameters:

Access-access files

Modify-modify the fil

Attrib-property change

Open-Open Fil

Delete-Delete a file

Create-New File

Move-File movement

-fromfile-read files that need to be monitored or excluded from files, one line per file, and excluded files starting with @

-timefmt-time format

-format-output format

-exclude-regular matching of files to be excluded, case sensitive

-excludei-regular matching of files to be excluded, ignoring case

% y%m%d% H% M-year, month, day clock

% T%w%f%e-time path file name status

5. Test 1

Check if the source directory has the following actions: modify,create,move,delete,attrib

Once it happens, publish to the target machine.

The mode is sshsrc: 192.168.22.11 (Rsync + Inotify-tools) dest: 192.168.22.12

Two machines need to do ssh secret-free login.

# mdkir / data/test/dest/-- dest machine # mdkir / data/test/src/-- src machine # rsync-av-- delete / data/test/src/ 192.168.22.12:/data/test/dest-- Test the command # vim / data/test/test.sh #! / bin/bash / usr/local/inotify/bin/inotifywait-mrq-e modify,create,move,delete Attrib / data/test/src | while read events do rsync-a-- delete / data/test/src/ 192.168.22.12:/data/test/dest echo "`date +'% F% T'` event: $events" > / tmp/rsync.log 2 > & 1 done # chmod 755 / data/test/test.sh # / data/test/test.sh & # echo'/ data/test/test.sh & > > / etc/rc.local-- set self-boot

We can also write a script like this on the target machine:

Rsync-a-delete / data/test/dest/ 192.168.22.11:/data/test/src

This enables two-way synchronization.

This is the answer to the question on how to understand the Linux remote data synchronization tool. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel for more related knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.