Shulou(Shulou.com)06/01 Report--

This article introduces how to use rsync to achieve file incremental synchronization in linux, the content is very detailed, interested friends can refer to, hope to be helpful to you.

1. System environment:

Update source server: 192.0.2.20

Destination server: 192.0.2.21

2. Destination server configuration: 192.0.2.21 (rsync server):

1. Check whether rsync is installed

Rpm-qa | grep rsync

2. Define rsync configuration file / etc/rsyncd.conf

192.0.2.21:

Cat > > / etc/rsyncd.conf > / etc/www1.pwd

And we need to set the permissions for this file to 600

Chmod 600 / etc/www1.pwd

Chmod 600 / etc/rsyncd.conf

5. Create a motd file (optional)

Rsyncd.motd records welcome messages for rsync services, in which you can enter any text information, such as:

Echo "Welcome to use the rsync services!" > > / var/rsyncd.motd

6. Start rsync

/ usr/bin/rsync-- daemon

Stop it

# / etc/init.d/rsyncd stop

Add to Auto start

Echo "/ usr/bin/rsync-- daemon" > > / etc/rc.local

Update the source server configuration: 192.0.2.20 (rsync client)

1. File system events that inotify can monitor include:

IN_ACCESS, that is, the file is accessed

IN_MODIFY, the file is write

IN_ATTRIB, file attributes are modified, such as chmod, chown, touch, etc.

IN_CLOSE_WRITE, writable files are close

IN_CLOSE_NOWRITE, unwritable files are close

IN_OPEN, the file is open

IN_MOVED_FROM, files are removed, such as mv

IN_MOVED_TO, files are moved, such as mv, cp

IN_CREATE, create a new file

IN_DELETE, files are deleted, such as rm

IN_DELETE_SELF, self-deletion, that is, an executable file deletes itself during execution

IN_MOVE_SELF, self-moving, that is, an executable file moves itself during execution

IN_UNMOUNT, the host file system is umount

IN_CLOSE, the file is closed, which is equivalent to (IN_CLOSE_WRITE | IN_CLOSE_NOWRITE)

IN_MOVE, the file is moved, which is equivalent to (IN_MOVED_FROM | IN_MOVED_TO)

Note: the files mentioned above also include directories.

2. Update the source server to install inotify-tools

Before installing inotify-tools, please make sure that your linux kernel reaches 2.6.13 and the CONFIG_INOTIFY option is turned on during compilation. You can also detect it by using the following command

Ls / proc/sys/fs/inotify

If there are three items of max_queued_events,max_user_instances,max_user_watches, it means support.

Download software: 3.14 is the last version of 2010

Wget http://cloud.github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz

Tar xvf inotify-tools-3.14.tar.gz

Mv inotify-tools-3.14 / opt/

Cd / opt/inotify-tools-3.14/

. / configure

Make;make install

After installing inotify-tools, the following two files are generated in the relevant installation directory:

Ll / usr/local/bin/

-rwxr-xr-x 1 root root 44327 Oct 10 15:32 inotifywait

-rwxr-xr-x 1 root root 41417 Oct 10 15:32 inotifywatch

Indicates that the installation was successful.

Note: installation is required on the source server, but not on the target server.

3. Write rsync monitoring script

Vi / root/rsync.sh

#! / bin/bash

Host1=192.0.2.21

Src=/data/www/

Des1=web1

User1=root

/ usr/local/bin/inotifywait-mrq-- timefmt'% d/%m/%y% HVA% M'-- format'% T% w% f'-e modify,delete,create,attrib ${src} | while read file

Do

Rsync-avzP-- delete-- progress ${src} ${user1} @ ${host1}:: ${des1}-password-file=/etc/www1.pwd & &

Echo "${file} was rsynced" > > / tmp/rsync.log 2 > & 1

Echo "-"

Done

Parameter description:

-m, that is,-- monitor, means to keep the event listening state at all times.

-r, that is,-- recursive, represents the recursive query directory.

-Q, that is,-- quiet, means to print out monitoring events.

-e, namely-- event. This parameter allows you to specify the events to be monitored. Common events include modify, delete, create, attrib, etc.

-- timefmt: the output format of the specified time

-- format: specify the details of the change file

Establish authentication file (only add password for rsync client authentication file)

Echo "root" > > / etc/www1.pwd

Chmod 600 / etc/www1.pwd

/ bin/sh-n / root/rsync.sh / / Syntax check

Chmod + x / root/rsync.sh

Start

/ root/rsync.sh

The error "/ usr/local/bin/inotifywait: error while loading shared libraries: libinotifytools.so.0" occurs

Solution:

32-bit system: ln-s / usr/local/lib/libinotifytools.so.0 / usr/lib/libinotifytools.so.0

64-bit system: ln-s / usr/local/lib/libinotifytools.so.0 / usr/lib64/libinotifytools.so.0

Put it backstage

Nohup sh / root/rsync.sh &

Set up automatic start

Echo "nohup sh / root/rsync.sh &" > > / etc/rc.local

IV. Synchronous testing

Create a new file on the update source server and run the following command to see if the files can be synchronized properly and to see if there is an error message

Rsync-avzP-- delete-- progress / data/www/ root@192.0.2.21::web1-- password-file=/etc/www1.pwd

Submit the updated files to the update source server, so that the updated files are synchronized to all destination servers in batches through inotify+rsync, which is quite convenient and quick.

5. Parameters related to inotify

Inotify defines the following interface parameters that can be used to limit the size of kernel memory consumed by inotify. Because these parameters are memory parameters, they can be adjusted in real time according to the application requirements:

/ proc/sys/fs/inotify/max_queued_evnets represents the maximum number of event that can be queued in the inotify instance when inotify_init is called. Events beyond this value are discarded, but the IN_Q_OVERFLOW event is triggered.

/ proc/sys/fs/inotify/max_user_instances represents the upper limit of the number of inotify instatnces that can be created per real user ID.

/ proc/sys/fs/inotify/max_user_watches represents the maximum number of directories that can be monitored per inotify instatnces. If the number of files being monitored is large, you need to increase the size of this value as appropriate.

Based on the above, it can be performed on either 32-bit or 64-bit systems:

Echo 104857600 > / proc/sys/fs/inotify/max_user_watches

Echo 'echo 104857600 > / proc/sys/fs/inotify/max_user_watches' > > / etc/rc.local

Http://blog.chinaunix.net/uid-25150840-id-5767502.html

=

Rsync with the use of crontab

1 installation

Check to see if rsync is installed

Install what is not installed.

Two servers:

192.0.2.20 Source server has directory / upload

192.0.2.21 Target server has directory / upload

2 create a rsyncd.conf file

Server-side creation of rsyncd.conf files

# vi / etc/rsyncd.conf

# [global]

Uid = root

Gid = root

Use chroot = no

Max connections = 10

List = yes

Pid file = / var/run/rsyncd.pid

Lock file = / var/run/rsyncd.lock

Log file = / var/log/rsyncd.log

Hosts allow = 192.0.2.21

# the IP allowed to be accessed is the client-side ip

[data]

# specify the publishing name, which is required when the client calls rsync

Path = / upload

# the release path is the directory you want to back up, which will be backed up to the client

Read only = no

Ignore errors

Auth users = root

# authenticate the user as root

Secrets file = / etc/sery.pass

# password file

Save exit

/ / start the rsync server

# rsync-daemon-config=/etc/rsyncd.conf

3 password

Generate client and server password files

Client:

# vi / etc/sery_client.pass

123456

# chmod 600 / etc/sery_client.pass

Server:

# attention. There is a user name on the server, a password after the client, and a permission of 600. so is the previous conf file.

# vi / etc/sery.pass

Root:123456

# chmod 600 / etc/sery.pass

4 start rsync

Server-side startup

# rsync-daemon-config=/etc/rsyncd.conf

5 View the monitoring status

It is also the server side.

# lsof-iRV 873

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

Rsync 15187 root 4U IPv4 83885 0t0 TCP *: rsync (LISTEN)

Rsync 15187 root 5u IPv6 83886 0t0 TCP *: rsync (LISTEN)

# netstat-antp | grep rsync

Tcp 0 0 0.0.0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0

Tcp 0 0: 873: * LISTEN 15187/rsync

6 write rsync to server self-startup

Or the server side?

# echo "/ usr/bin/rsync-- daemon-- config=/etc/rsyncd.conf" > > / etc/rc.local

7 Test synchronization

Client execution

# rsync-avzP-delete root@192.0.2.20::data / upload-password-file=/etc/sery_client.pass

1.-a uses archive mode to maintain the original file permissions

2.-v is displayed on the screen

3.-Z compress data during transmission

4.-P transmission progress

5.-- delete means that if the server deletes the file, then the client deletes the file accordingly to maintain true consistency

6. Data is the release name mentioned earlier, and / root/test client local storage path

7.-password-file specified password file

To synchronize hard links, add-H

8 client configuration scheduling task crontab

Write crontab on the client

# vi / root/rsyncd.sh

#! / bin/bash

Rsync-avzP-delete root@192.168.1.64::data / root/test/data-password-file=/etc/sery_client.pass

# wq

Check to see if synchronization is normal.

The following items are tested and the synchronization is normal:

Create a file

Modify a file

Delete a file

Create a folder

Create a file in a folder

9 / other

(1) you can also use ssh links instead of password when connecting between client and server.

Automatic ssh/scp method = =

An is the local host (that is, the machine used to control other hosts); 192.0.2.21

B is the remote host (that is, the controlled machine Server), 192.0.2.20

Both An and B's systems are Linux.

Run the command on A:

# ssh-keygen-t rsa (enter for three times in a row, that is, the public key and private key are generated locally without setting a password)

# ssh root@192.0.2.20 "mkdir .ssh" (password required)

# scp ~ / .ssh/id_rsa.pub root@192.0.2.20:.ssh/id_rsa.pub (password required)

Command on B:

# touch / root/.ssh/authorized_keys (skip this if this file already exists)

# cat / root/.ssh/id_rsa.pub > > / root/.ssh/authorized_keys (append the contents of id_rsa.pub to authorized_keys)

Go back to machine A:

# ssh root@192.0.2.20 (No password required, login is successful)

A simpler way

Open the terminal to execute ssh-keygen, and this command will create two files, public key and private key, in the ~ / .ssh / directory: id_rsa and id_rsa.pub.

Just copy the public key to the server's ~ / .ssh/authorized_keys file. The methods are as follows:

Cat ~ / .ssh/id_rsa.pub | ssh-p 22 root@host 'cat > > ~ / .ssh/authorized_keys'

=

Three / commands about rsync

Rsync commands can be in the following six formats:

Rsync [OPTION]... SRC DEST

Rsync [OPTION]... SRC [USER@] HOST:DEST

Rsync [OPTION]... [USER@] HOST:SRC DEST

Rsync [OPTION]... [USER@] HOST::SRC DEST

Rsync [OPTION]... SRC [USER@] HOST::DEST

Rsync [OPTION]... Rsync:// [USER@] HOST [: PORT] / SRC [DEST]

Rsync has six different modes of operation corresponding to the above six command formats:

1) copy the local file. This mode of operation is started when neither SRC nor DES path information contains a single colon: delimiter. For example: rsync-a / data / backup

2) use a remote shell program (such as rsh, ssh) to copy the contents of the local machine to the remote machine. Start this mode when the DST path address contains a single colon ":" separator. Such as: rsync-avz * .c foo:src

3) use a remote shell program (such as rsh, ssh) to copy the contents of the remote machine to the local machine. Start this mode when the SRC address path contains a single colon ":" separator. For example: rsync-avz foo:src/bar / data

4) copy files from the remote rsync server to the local machine. This mode is started when the SRC path information contains the "::" delimiter. For example: rsync-av root@172.16.78.192::www / databack

5) copy files from the local machine to the remote rsync server. This mode is started when the DST path information contains the "::" delimiter. For example: rsync-av / databack root@172.16.78.192::www

6) list the files of the remote machine. This is similar to rsync transport, except that the local machine information is omitted from the command. Such as: rsync-v rsync://172.16.78.192/www

The rsync parameter is explained as follows:

-v,-- verbose verbose mode output

-Q,-- quiet compact output mode

-c,-- checksum turns on the check switch to force the file transfer to be checked

-a,-- archive archiving mode, which means that files are transferred recursively and all file attributes are maintained, which is equal to-rlptgoD

-r,-- recursive processes subdirectories in recursive mode

-R,-- relative uses relative path information

-b,-- backup creates a backup, that is, rename the old file to ~ filename when the same file name already exists for the destination. You can use the-- suffix option to specify different backup file prefixes.

-- backup-dir stores backup files (such as ~ filename) in a directory.

-suffix=SUFFIX defines the backup file prefix

-u,-- update only updates, that is, skips all files that already exist in DST and the file time is later than the file to be backed up. (do not overwrite updated files)

-l,-- links retains soft links

-L,-- copy-links handles soft links like regular files

-- copy-unsafe-links only copies links that point outside the SRC path directory tree

-- safe-links ignores links that point outside the SRC path directory tree

-H,-- hard-links retains hard links

-p,-- perms maintains file permissions

-o,-- owner keeps file owner information

-g,-- group keeps file group information

-D,-- devices keeps device file information

-t,-- times keeps file time information

-S,-- sparse performs special processing on sparse files to save DST space

-n,-- dry-run reality which files will be transferred

-W,-- whole-file copies files without incremental detection

-x,-- one-file-system do not cross file system boundaries

-B,-- the block size used by the block-size=SIZE check algorithm. The default is 700byte.

-e,-- rsh=COMMAND specifies to use rsh and ssh methods for data synchronization

-- rsync-path=PATH specifies the path information where the rsync command is located on the remote server

-C,-- cvs-exclude automatically ignores files using the same method as CVS to exclude files that you don't want to transfer

-- existing updates only those files that already exist in DST, not those newly created files

-- delete deletes files in DST that are not available in SRC

-- delete-excluded also deletes files on the receiver that are specified and excluded by this option

-- delete delete-after after the transfer is finished.

Delete ignore-errors even if there are IO errors.

-- max-delete=NUM deletes a maximum of NUM files

-- partial retains files that were not fully transferred for some reason, thus speeding up subsequent retransfers

-- force forces directory deletion, even if it is not empty

-- numeric-ids does not match numeric user and group ID to username and group name

-- timeout=TIME IP timeout in seconds

-I,-- ignore-times does not skip files of the same time and length

-- size-only when deciding whether to back up a file, only look at the file size without considering the file time

-- the timestamp window used by modify-window=NUM to determine whether the file is at the same time. The default is 0.

-T-- temp-dir=DIR creates temporary files in DIR

-- compare-dest=DIR also compares files in DIR to decide whether a backup is needed

-P equals-- partial

-- progress shows the backup process

-z,-- compress compresses the backed-up files during transfer

-- exclude=PATTERN specifies to exclude files that do not need to be transferred

-- include=PATTERN specifies the file mode that needs to be transferred without exclusion

-- exclude-from=FILE excludes files of the specified pattern in FILE

-- include-from=FILE does not exclude files specified by FILE for pattern matching

-- version print version information

-- address is bound to a specific address

-- config=FILE specifies other configuration files and does not use the default rsyncd.conf file

-- port=PORT specifies other rsync service ports

-- blocking-io uses blocking IO for remote shell

-stats gives the transfer status of some files

-- the actual transmission process of progress during transmission

-- log-format=formAT specifies the log file format

-- password-file=FILE gets the password from FILE

-- bwlimit=KBPS restricts the bandwidth of IBO, KBytes per second

-h,-- help displays help information

On how to use rsync in linux to achieve file incremental synchronization is shared here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.