Shulou(Shulou.com)06/01 Report--

This article mainly introduces "how to improve the copying efficiency of large files by Linux". In daily operation, I believe many people have doubts about how to improve the copying efficiency of large files by Linux. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts of "how to improve the copying efficiency of large files by Linux". Next, please follow the editor to study!

You can copy from the source machine to the target machine one by one, but the time is often multiplied. Or you can copy from the source machine to multiple target machines at the same time, but the speed is not really fast due to factors such as the bandwidth of the source machine.

Fortunately, you can do better with some UNIX tools. Combine tee and FIFO to form a fast file distribution chain: each machine in the distribution chain saves the file and distributes it to its next link.

First, select a target machine as the last link of the distribution chain, on which you only need to listen with nc (assuming the port is 1234), then decompress it by pigz through the pipeline, and continue to decompose the data by tar through the pipeline.

Nc-l 1234 | pigz-d | tar xvf-

Then, go up from the end of the distribution chain and set up other target machines, which also need to be monitored, decompressed and decomposed, but before decompression, we use the tee command to output the data to the named pipe (FIFO), and another shell pipe will distribute the undecompressed data to the next link of the distribution chain:

Mkfifo myfifo

Nc hostname_of_next_box 1234 nc-l 1234 | tee myfifo | pigz-d | tar xvf-

Finally, start the distribution chain on the source machine to transfer the data to the first ring of the distribution chain:

Tar cv some_files | pigz | nc hostname_of_first_box 1234

In my tests, each machine in the distribution chain lost about 3% of the performance (compared to 1-to-1 copies), but compared to one copy one by one or a single machine distributed to multiple machines at the same time, the efficiency improvement is obvious.

At this point, the study on "how to improve the copy efficiency of large files by Linux" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.