Perl multithreaded rsync backup files to the remote host

Demand:

There are hundreds of gigabytes of backup files on the host to rsync to the remote host, and we cut the large files into dozens of small files for multi-thread transfer.

Here is a demonstration using 14 1G files:

[root@vm0 test] # pwd/root/test [root@vm0 test] # ll total consumption 13631540 RW / RW / R / M / R / R / M / R / R 1 root root 1073741824 June 11 18:29 test10.data-rw-r--r--. 1 root root 1073741824 June 11 18:30 test11.data-rw-r--r--. 1 root root 1073741824 June 11 18:31 test12.data-rw-r--r--. 1 root root 1073741824 June 11 18:31 test13.data-rw-r--r--. 1 root root 1073741824 June 11 18:32 test14.data-rw-r--r--. 1 root root 1073741824 June 11 18:23 test2.data-rw-r--r--. 1 root root 1073741824 June 11 18:24 test3.data-rw-r--r--. 1 root root 1073741824 June 11 18:25 test4.data-rw-r--r--. 1 root root 1073741824 June 11 18:26 test5.data-rw-r--r--. 1 root root 1073741824 June 11 18:26 test6.data-rw-r--r--. 1 root root 1073741824 June 11 18:27 test7.data-rw-r--r--. 1 root root 1073741824 June 11 18:28 test8.data-rw-r--r--. 1 root root 1073741824 June 11 18:29 test9.data [root@vm0 test] #

Script name: tq.pl

#! / usr/bin/env perluse strict;use threads;use Thread::Queue;use File::Find;use File::Rsync;use POSIX qw (strftime); # Local host file directory my $srcFilePath='/root/test/';# uses queues to insert the files to be backed up one by one into the queue my $fileQueue = Thread::Queue- > new (); # remote host backup directory my $remotedir='lansgg@192.168.137.129::lansggtest';# maximum number of threads my $thread_max = 5 My $backupTime = strftime ("% Y%m%d%H%M%S", localtime (time)); print "begin: $backupTime\ n"; # retrieve all files in the directory to be backed up,. Except for. In linux. Represents the current directory sub findAllFile {unless ($_ eq'.') {print "corrent file: $File::Find::name\ n"; $fileQueue- > enqueue ($_);}} find (\ & findAllFile,$srcFilePath); # transfer sub rsync {my $file = shift; print "rsync-- $file\ n" using rsync My $obj = File::Rsync- > new ({archive = > 1, compress = > 1, checksum = > 1, recursive = > 1, times = > 1 # verbose = > 1, timeout = > 300, progress = > 1, stats = > 1, 'ignore-times' = > 1,' password-file' = >'. / rsync.pass',}); $obj- > exec ({src = > "$srcFilePath$file", dest = > $remotedir}) or warn "rsync Failed!\ n" # print $obj- > out;} # check for untransferred files in the queue while ($fileQueue- > pending ()) {if (threads- > list ())

< $thread_max ){ my $readQueue = $fileQueue->

Dequeue (); # print "current file Queue is $readQueue\ n"; # generate thread threads- > create (\ & rsync,$readQueue); # View the total number of current threads my $thread_count = threads- > list (); # print "thread_count is $thread\ n" } # determine whether the current thread completes the job, and recycle foreach my $thread (threads- > list (threads::all)) {if ($thread- > is_joinable ()) {$thread- > join () }} # join drops the remaining threads (because there may still be threads executing when the queue in while is empty, but at this time the program will exit the while loop, so an additional program join the remaining threads) foreach my $thread (threads- > list (threads::all)) {$thread- > join ();} $backupTime = strftime ("% Y%m%d%H%M%S", localtime (time)) Print "end: $backupTime\ n"

This script uses the core functions, which can be added at a later stage, such as logging, mail sending and so on.

When we execute the script, look at the host thread

[root@vm0 pl] # ps-ef | grep tqroot 6377 2152 88 19:05 pts/3 00:00:12 perl. / tq.pl [root@vm0 pl] # pstree-p 6377perl (6377) ─┬─ rsync (6379) ├─ rsync (6381) ├─ rsync (6383) ├─ rsync (6385) ├─ rsync (6387) ├─ {perl} (6378) ├─ {perl} (6380) ├─ {perl} (6382) ├─ {perl} (6384) └─ {perl} (6386) [root@vm0 pl] # ps-ef | grep rsyncroot 6379 6377 14 19:05 pts/3 00:00:14 rsync-archive-checksum-compress-ignore-times-progress-recursive-stats-times -- password-file=./rsync.pass-- timeout=300 / root/test//test13.data lansgg@192.168.137.129::lansggtestroot 6381 6377 19:05 pts/3 00:00:14 rsync-- archive-- checksum-- compress-- ignore-times-- progress-- recursive-- stats-- times-- password-file=./rsync.pass-- timeout=300 / root/test//test12.data lansgg@192.168.137.129::lansggtestroot 6383 6377 14 19:05 pts/3 00:00:14 rsync-- archive-- checksum-- compress-- ignore-times-- progress-- recursive-- stats-- times-- password-file=./rsync.pass-- timeout=300 / root/test//test1.data lansgg@192.168.137.129::lansggtestroot 6385 6377 14 19:05 pts/3 00:00:14 rsync-archive-- checksum-compress-ignore-times-- progress-- recursive-- stats-- times -- password-file=./rsync.pass-timeout=300 / root/test//test8.data lansgg@192.168.137.129::lansggtestroot 6387 6377 12 19:05 pts/3 00:00:12 rsync-- archive-- checksum-- compress-- ignore-times-- progress-- recursive-- stats-- times-- password-file=./rsync.pass-- timeout=300 / root/test//test3.data lansgg@192.168.137.129::lansggtestroot 6399 2193 0 19:06 pts/2 00:00:00 grep rsync