Shulou(Shulou.com)06/02 Report--

This article mainly explains the method of quickly copying large files under linux, the content is clear, interested friends can learn, I believe it will be helpful after reading.

Copy data

When copying data remotely, we usually use the rsync command, but if we copy a large number of small files, the transfer speed of rsync will be slow. Using tar pv lz4 to package and compress the transfer can solve this problem. Using this method is equivalent to using scp and rsync to transfer large files.

Measured, the use of rsync transmission 1200G, a single file size of dozens of KB~2GB, gigabit network card, need to run 6 rsync at the same time to run full bandwidth, each speed 20MB or so, speed fluctuations. You can copy 4.5GB every minute or so.

However, with tar pv lz4, it is OK to run one, and the speed fluctuation is small. You can copy 6.8GB every minute or so.

Examples of rsync usage

Rsync installation: yum install-y rsync

# push [root@vm5] # rsync-auvzP-e "ssh-p22" mssh.tar.gz root@192.168.176.11:/data/sending incremental file listmssh.tar.gz 1977 100% 0.00kB/s 0:00:00 (xfr#1 To-chk=0/1) sent 2069 bytes received 35 bytes 4208.00 bytes/sectotal size is 1977 speedup is 0.9 pull [root@vm5 ~] # rm-f mssh.tar.gz [root@vm5 ~] # rsync-auvzP-e "ssh-p22" root@192.168.176.11:/data/mssh.tar.gz. Receiving incremental file listmssh.tar.gz 1977 100 1.89MB/s 0:00:00 (xfr#1 To-chk=0/1) sent 43 bytes received 2069 bytes 4224.00 bytes/sectotal size is 1977 speedup is 0.94

Parameter auvzP explains: parameter an is archived transfer, file attributes are retained, u is update transfer, and source file modification time is newer, then transfer. V is to show the detailed process, z is the compressed transmission, and P is the breakpoint transmission.

Note: when rsync transfers a folder, folder/ will transfer the files in the directory without / transferring the folder as well.

Use compressed transmission

Install pv, lz4 tools

Note: both ends of the server should be installed.

Pv is not in the yum source. You can find it on the official website of pv.

# go to the pv official website and get a link to the rpm package Direct rpm command to install [root@vm5 ~] # rpm- ivh http://www.ivarch.com/programs/rpms/pv-1.6.6-1.x86_64.rpm to get http://www.ivarch.com/programs/rpms/pv-1.6.6-1.x86_64.rpm warning: / var/tmp/rpm-tmp.mFbA6u: header V3 DSA/SHA1 Signature Key ID 3fc56f51: preparing for NOKEY. # # [100%] upgrading / installing... 1:pv-1.6.6-1 # # [100%] # lz4 can install [root@vm5] # yum install-y lz4 directly with yum

Use

[root@vm5 ~] # time tar-c go | pv | lz4-B4 | ssh-P22-c aes128-ctr 192.168.176.11 "lz4-d | tar-xC / data/" using blocks of size 64 KB18.1MiB 0:00:00 [49.5MiB/s] [] real 0m0.376suser 0m0.080ssys 0m0. 108s# compares rsync [root@vm5] # time rsync-auvzP-e "ssh-p22" go 192.168.176.11:/data/.sent 11741677 bytes received 10451 bytes 7834752.00 bytes/sectotal size is 18502481 speedup is 1.57real 0m1.130suser 0m0.797ssys 0m0.160s [root@vm5] # read the above Do you have a further understanding of the method of quickly copying large files under linux? if you want to learn more, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.