Shulou(Shulou.com)06/02 Report--

Sersync data synchronization

I. introduction to rsync

Rsync is a data mirroring backup tool-remote sync under unix-like systems. A fast incremental backup tool Remote Sync, remote synchronization supports local replication, or synchronization with other SSH, rsync hosts.

Its features are as follows:

V can mirror and save the entire directory tree and file system.

V can easily maintain the permissions, time, soft and hard links of the original file, and so on.

V can be installed without special permission.

V Fast: rsync copies everything the first time it synchronizes, but only transfers modified files the next time. Rsync can compress and decompress data during transmission, so less bandwidth can be used.

V security: you can use scp, ssh, etc., to transfer files, of course, you can also use a direct socket connection.

V supports anonymous transmission to facilitate website mirroring.

IP

Role

Install softwar

Catalogue

Purpose

192.168.91.166

Rsync daemon server

Rsync

/ data

Push the directory under / data on the 192.168.91.156 rsync client to the 192.168.91.166 rsync server / data in real time

192.168.91.156

Rsync client

Serync

/ data

II. Installation of rsync

Yum install-y rsync

III. Rsync sends data to the remote end through ssh authentication transfer 3.1

Rsync-avz-P local directory-e 'ssh-p port number' remote user name @ remote IP: remote path

Root@template / tmp 16:33:07 # cat rsync.txt

I'm the rsync test.

Root@template / tmp 16:34:02 # rsync-avzP / tmp/rsync.txt-e 'ssh-p 22' root@192.168.44.100:/tmp

The authenticity of host '192.168.44.100 (192.168.44.100)' can't be established.

RSA key fingerprint is57:22:33:d6:43:40:31:84:88:19:5d:f5:fc:7d:4e:ce.

Are you sure you want to continue connecting (yes/no)? Yes

Warning: Permanently added '192.168.44.100' (RSA) tothe list of known hosts.

Reverse mapping checking getaddrinfo for bogon [192.168.44.100] failed-POSSIBLE BREAK-IN ATTEMPT!

Root@192.168.44.100's password:

Sending incremental file list

Rsync.txt

18100% 0.00kB/s 0:00:00 (xfer#1, to-check=0/1)

Sent 93 bytes received 31 bytes 13.05 bytes/sec

Total size is 18 speedup is 0.15

Detailed parameters:

-v,-- verbose verbose mode output

-Q,-- quiet compact output mode

-c,-- checksum turns on the check switch to force the file transfer to be checked

-a,-- archive archiving mode, which means that files are transferred recursively and all file attributes are maintained, which is equal to-rlptgoD

-r,-- recursive processes subdirectories in recursive mode

-R,-- relative uses relative path information

-b,-- backup creates a backup, that is, rename the old file to ~ filename when the same file name already exists for the destination. You can use the-- suffix option to specify different backup file prefixes.

-- backup-dir stores backup files (such as ~ filename) in a directory.

-suffix=SUFFIX defines the backup file prefix

-u,-- update only updates, that is, skips all files that already exist in DST and the file time is later than the file to be backed up. (do not overwrite updated files)

-l,-- links retains soft links

-L,-- copy-links handles soft links like regular files

-- copy-unsafe-links only copies links that point outside the SRC path directory tree

-- safe-links ignores links that point outside the SRC path directory tree

-H,-- hard-links retains hard links

-p,-- perms maintains file permissions

-o,-- owner keeps file owner information

-g,-- group keeps file group information

-D,-- devices keeps device file information

-t,-- times keeps file time information

-S,-- sparse performs special processing on sparse files to save DST space

-n,-- dry-run reality which files will be transferred

-W,-- whole-file copies files without incremental detection

-x,-- one-file-system do not cross file system boundaries

-B,-- the block size used by the block-size=SIZE check algorithm. The default is 700byte.

-e,-- rsh=COMMAND specifies to use rsh and ssh methods for data synchronization

-- rsync-path=PATH specifies the path information where the rsync command is located on the remote server

-C,-- cvs-exclude automatically ignores files using the same method as CVS to exclude files that you don't want to transfer

-- existing updates only those files that already exist in DST, not those newly created files

-- delete deletes files in DST that are not available in SRC

-- delete-excluded also deletes files on the receiver that are specified and excluded by this option

-- delete delete-after after the transfer is finished.

Delete ignore-errors even if there are IO errors.

-- max-delete=NUM deletes a maximum of NUM files

-- partial retains files that were not fully transferred for some reason, thus speeding up subsequent retransfers

-- force forces directory deletion, even if it is not empty

-- numeric-ids does not match numeric user and group ID to username and group name

-- timeout=TIME IP timeout in seconds

-I,-- ignore-times does not skip files of the same time and length

-- size-only when deciding whether to back up a file, only look at the file size without considering the file time

-- the timestamp window used by modify-window=NUM to determine whether the file is at the same time. The default is 0.

-T-- temp-dir=DIR creates temporary files in DIR

-- compare-dest=DIR also compares files in DIR to decide whether a backup is needed

-P equals-- partial

-- progress shows the backup process

-z,-- compress compresses the backed-up files during transfer

-- exclude=PATTERN specifies to exclude files that do not need to be transferred

-- include=PATTERN specifies the file mode that needs to be transferred without exclusion

-- exclude-from=FILE excludes files of the specified pattern in FILE

-- include-from=FILE does not exclude files specified by FILE for pattern matching

-- version print version information

-- address is bound to a specific address

-- config=FILE specifies other configuration files and does not use the default rsyncd.conf file

-- port=PORT specifies other rsync service ports

-- blocking-io uses blocking IO for remote shell

-stats gives the transfer status of some files

-- the actual transmission process of progress during transmission

-- log-format=formAT specifies the log file format

-- password-file=FILE gets the password from FILE

-- bwlimit=KBPS restricts the bandwidth of IBO, KBytes per second

-h,-- help displays help information

3.2 pull data from the remote end

Rsync-avz-P-e 'ssh-p Port number' remote user name @ remote IP: remote path local directory

Root@template / tmp 16:34:19 # rsync-avzP-e'ssh-p 22'root@192.168.44.100:/tmp/rsync.txt / tmp

Reverse mapping checking getaddrinfo for bogon [192.168.44.100] failed-POSSIBLE BREAK-IN ATTEMPT!

Root@192.168.44.100's password:

Receiving incremental file list

Rsync.txt

19100% 18.55kB/s 0:00:00 (xfer#1, to-check=0/1)

Sent 36 bytes received 99 bytes 30.00 bytes/sec

Total size is 19 speedup is 0.14

Root@template / tmp 16:39:26 # cat rsync.txt

I'm rsync Test 2.

four。 Backup server configuration via rsync authentication transport 4.1, that is, server-side 192.168.91.1664.1.1 create rsync synchronization user [root@salt-master data] # useradd-M-s / sbin/nologin rsync4.1.2 modify configuration file (vim / etc/rsyncd.conf)

[root@salt-master data] # cat / etc/rsyncd.conf

# config_start

# # rsyncd.conf start##

Uid = root # the user can be something else at this time, but as long as you have the permission to manage / data on the server, you don't have to use root.

Gid = root

Use chroot = no # the private network is set to no

Max connections = 200 # the maximum number of connections defaults to 0 infinity, while the negative number turns off

Timeout = 300 # timeout defaults to 0 notimeout No timeout is recommended 200600 (5-10 minutes)

Pid file = / var/run/rsyncd.pid

Lock file = / var/run/rsync.lock

Log file = / var/log/rsyncd.log

[data] # New module name

Comment = datafile by caimengzhi at 2016-11-10 # comment section

Path = / data/ # default upload and download paths

Ignore errors # ignore errors

Read only = false # disable read-only writeability

List = false # forbidden list

Hosts allow = 192.168.91.0 hosts allow 24 # specifies the client ip or address field that can be connected

Hosts deny = 0.0.0.0amp 32 # specifies an ip or address field that cannot be connected

Auth users = rsync_body # virtual user synchronized by the client

Secrets file = / etc/rsync.password # password of the virtual machine user synchronized by the client

4.1.3 add a password file

[root@salt-master data] # echo 'rsync_body:admin' > > / etc/rsync.password # add an account and password, which are the virtual accounts and passwords for synchronization.

[root@salt-master data] # cat / etc/rsync.password

Rsync_body:admin

[root@salt-master data] # chmod 600 / etc/rsync.password

4.1.4 start the service

[root@salt-master data] # rsync-- daemon # rsync server starts in daemon mode and port is 873 of tcp protocol

[root@salt-master data] # ps-ef | grep rsync | grep-v grep

Root 7247 1 0 17:21? 00:00:00 rsync-daemon

[root@salt-master data] # netstat-lnp | grep 873

Tcp 0 0 0.0.0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0

Tcp 0 0: 873: * LISTEN 7247/rsync

[root@salt-master data] # lsof-iRU 873

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

Rsync 7247 root 4U IPv4 67065 0t0 TCP *: rsync (LISTEN)

Rsync 7247 root 5u IPv6 67066 0t0 TCP *: rsync (LISTEN)

4.2 Source data server configuration, that is, client 192.168.91.1564.2.1 configuration password file

[root@salt-client01 sersync] # echo "admin" > / etc/rsync.password # client only needs to configure the password file, because the virtual synchronization account is specified during synchronization

[root@salt-client01 sersync] # cat / etc/rsync.password

Admin

[root@salt-client01 sersync] # chmod 600 / etc/rsync.password

[root@salt-client01 sersync] # ll / etc/rsync.password

-rw- 1 root root 6 Nov 10 17:26 / etc/rsync.password

4.2.2 send data to the backup server

Rsync-avz-P local directory or file rsync_backup@ server address:: module name-password-file=/etc/rsync.pwd

[root@salt-client01 sersync] # rsync-avz / data rsync_body@192.168.91.166::data-- password-file=/etc/rsync.password

Sending incremental file list

Data/

Data/3

Sent 88 bytes received 31 bytes 238.00 bytes/sec

Total size is 0 speedup is 0.00

# indicates that rsync server and client are OK

4.2.3 pull from the backup server

Rsync-avz-P rsync_backup@ server address:: module name local directory or file-password-file=/etc/rsync.pwd

5. The simple principle of sersync

Using inotify to monitor files, when monitoring files have changed, it will call rsync to achieve trigger real-time synchronization!

Install sersync (note that sersync works on the source server of rsync, that is, on the client)

[root@salt-client01 ~] # cd / usr/local/src/

[root@salt-client01 src] # tar-zxvf sersync2.5.4_64bit_binary_stable_final.tar.gz-C / usr/local/

GNU-Linux-x86/

GNU-Linux-x86/sersync2

GNU-Linux-x86/confxml.xml

[root@salt-client01 src] # cd / usr/local/

[root@salt-client01 local] # mv GNU-Linux-x86 sersync

[root@salt-client01 local] # cd sersync/

[root@salt-client01 sersync] # mkdir conf bin log

[root@salt-client01sersync] # mv confxml.xml conf

[root@salt-client01sersync] # mv sersync2 bin/sersync

5.2 modify the configuration file

* * start with 30 lines * *

#-artuz is the parameter for rsync synchronization

* * Line 36 * *

*

* Note: if there are multiple directory backups, you can add multiple configuration files to the-o parameter at startup.

[root@salt-client01 conf] # diff confxml.xml confxml.xml.bak

24,25c24,25

< #data就是本地需要同步的文件夹到服务器端的目录 < #data (server的模块名)是rsync 服务端的文件夹，也就是推送到服务器端的目标文件夹，可以配置多个， --- >

>

31c31

< #true 才能生效，rsync_body同步时候虚拟账号，后面是密码文件 --- >

33c33

< #true 才能生效 --- >

36c36

< #检测rsync进程判断，没有自动启 --- >

5.3 start sersync

[root@salt-client01 src] # echo 'export PATH=$PATH:/usr/local/sersync/bin' > > / etc/profile

# declare environment variables

[root@salt-client01 src] # source / etc/profile

[root@salt-client01 src] # sersync2-r-d-o / usr/local/sersync/conf/confxml.xml # launch

Set the system param

Execute:echo 50000000 > / proc/sys/fs/inotify/max_user_watches

Execute:echo 327679 > / proc/sys/fs/inotify/max_queued_events

Parse the command param

Option:-r rsync all the local files to the remote servers before the sersync work

Option:-d run as a daemon

Option:-o config xml name: / usr/local/sersync/conf/confxml.xml

Daemon thread num: 10

Parse xml config file

Host ip: localhosthost port: 8008

Daemon start,sersync run behind the console

Use rsync password-file:

User isrsync_body

Passwordfile is / etc/rsync.password

Config xml parse success

Please set / etc/rsyncd.conf max connections=0 Manually

Sersync working thread 12 = 1 (primary thread) + 1 (fail retry thread) + 10 (daemon sub threads)

Max threads numbers is: 22 = 12 (Thread pool nums) + 10 (Sub threads)

Please according your cpu, use-n param to adjust the cpu rate

-

Rsync the directory recursivly to the remote servers once

Working please wait...

Execute command: cd / data & & rsync-artuz-R-delete. /-- timeout=100 rsync_body@192.168.91.166::data-- password-file=/etc/rsync.password > / dev/null 2 > & 1

Run the sersync:

Watch path is: / data # at this point you can see that sersync has started successfully

# Test script

[root@salt-client01 log] # pwd

/ usr/local/sersync/log

[root@salt-client01 log] # vim rsync_fail_log.sh

[root@salt-client01 log] # chmod + x rsync_fail_log.sh

[root@salt-client01 ~] # cat / usr/local/sersync/log/rsync_fail_log.sh

#! / bin/bash

# Purpose: Check sersync whether it is alive

# Author: cai meng zhi

SERSYNC= "/ usr/local/sersync/bin/sersync2"

CONF_FILE= "/ usr/local/sersync/conf/confxml.xml"

STATUS=$ (ps aux | grep' sersync2' | grep-v 'grep' | wc-l)

If [$STATUS-eq 0]

Then

$SERSYNC-d-r-o $CONF_FILE &

Else

Exit 0

Fi

After the script is written, add it to the scheduled task

* / 1 * / bin/bash / usr/local/sersync/log/rsync_fail_log.sh > / dev/null 2 > & 1

Test synchronization:

Add files to the client

[root@salt-client01 data] # cp / etc/passwd 192.168.91.156.passwd

[root@salt-client01 data] # ll

Total 4

-rw-r--r-- 1 root root 1928 Nov 10 18:15 192.168.91.156.passwd

-rw-r--r-- 1 root root 0 Nov 10 17:27 3

Server detection

[root@salt-master data] # cd / data/

[root@salt-master data] # ll

Total 8

-rw-r--r-- 1 root root 1928 Nov 10 18:15 192.168.91.156.passwd # indicates that it has been synchronized

-rw-r--r-- 1 root root 0 Nov 10 17:27 3

Drwxr-xr-x 2 root root 4096 Nov 10 17:27 data

Client test deletion

[root@salt-client01 data] # rm rf 192.168.91.156.passwd

Rm: cannot remove `rf': No such file or directory

Rm: remove regular file `192.168.91.156.passwdflowers? Y

[root@salt-client01 data] # ll

Total 0

-rw-r--r-- 1 root root 0 Nov 10 17:27 3

[root@salt-client01 data] #

Server side:

[root@salt-master data] # ll

Total 4

-rw-r--r-- 1 root root 0 Nov 10 17:27 3 # indicates that it has been deleted

Drwxr-xr-x 2 root root 4096 Nov 10 17:27 data

5.4 Summary of common errors

Error one:

@ ERROR: auth failed on module xxxxx

Rsync: connection unexpectedly closed (90 bytes read so far)

Rsync error: error in rsync protocoldata stream (code 12) at io.c

Note: this is because the password is set incorrectly and you cannot log in successfully. Check rsync.pwd to see if the customer service matches. This can also happen if the rsync service is not started on the server side.

Error 2:

Password file must not beother-accessible

Continuing without password file

Password:

Note: this is because the permission of rsyncd.pwdrsyncd.sec is wrong and should be set to 600. Such as: chmod600 rsyncd.pwd

Error 3:

@ ERROR: chroot failed

Rsync: connection unexpectedly closed (75 bytes read so far)

Rsync error: error in rsync protocoldata stream (code 12) at io.c

Note: this is because the path path you set in rsync.conf does not exist. You need to create a new directory to enable synchronization.

Error 4:

Rsync: failed to connect to218.107.243.2: No route to host

Rsync error: error in socket IO (code10) at clientserver.c [receiver=2.6.9]

Explanation: firewall problems lead to, it is best to completely shut down the firewall, the wrong basic law is like this, whether it is S or C, as well as ignore errors option issues will also lead to

Error five:

@ ERROR: access denied to www fromunknown (192.168.1.123)

Rsync: connection unexpectedly closed (0 bytes received so far) [receiver]

Rsync error: error in rsync protocoldata stream (code 12) at io.c

Description: this problem is obvious, it is the configuration option hostallow, beginners like to allow a segment to make a configuration, and then the module is the same, resulting in

Error 6:

Rsync error: received SIGINT,SIGTERM, or SIGHUP (code 20) at rsync.c [generator=2.6.9]

Rsync error: received SIGUSR1 (code19) at main.c (1182) [receiver=2.6.9]

Description: this problem is mostly caused by the server service is not started normally, go to the server to check whether the service is started, and then check whether the / var/run/rsync.pid file exists, the most straightforward way is to kill the service that has been started, and then start the service again or let the script join the system startup service level and then shutdown-r now server

Error 7:

Rsync: read error: Connection resetby peer

Rsync error: error in rsync protocoldata stream (code 12) at io.c [sender=2.6.9]

Description: no data exists in the original data directory

six。 Data

.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.