Shulou(Shulou.com)06/01 Report--

This article mainly explains "how to speed up the Linux command". The content of the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "how to accelerate the Linux command".

We all know that grep, bzip2, wc, awk, sed, etc., are all single-threaded and can only use one CPU kernel. So how can I use these kernels?

For the Linux command to use all the CPU kernels, we need to use the GNU Parallel command, which allows all of our CPU kernels to do magical map-reduce operations in a single machine, of course, with the rarely used-pipes parameter (also known as-spreadstdin). In this way, your load will be evenly distributed among the CPU, really.

BZIP2

Bzip2 is a better compression tool than gzip, but it's slow! Stop messing around, we have a way to solve the problem.

Previous practice:

Cat bigfile.bin | bzip2-- best > compressedfile.bz2

Now it goes like this:

Cat bigfile.bin | parallel-- pipe-- recend'--k bzip2-- best > compressedfile.bz2

Especially for bzip2,GNU parallel, it is super fast on multicore CPU. As soon as you are not careful, it will be finished.

GREP

If you have a very large text file, you might have done this before:

Grep pattern bigfile.txt

Now you can do this:

Cat bigfile.txt | parallel-- pipe grep 'pattern'

Or this:

Cat bigfile.txt | parallel-- block 10m-- pipe grep 'pattern'

This second use uses the-block 10m parameter, which means that each kernel processes 10 million rows-- you can use this parameter to adjust how many rows each CUP kernel processes.

AWK

Here is an example of using the awk command to calculate a very large data file.

General usage:

Cat rands20M.txt | awk'{print s} 1} END {slots}'

Now it goes like this:

Cat rands20M.txt | parallel-- pipe awk\'{slots =\ $1} END {print s}\'| awk'{print s} 1} END {slots}'

This is a bit complicated: the-pipe parameter in the parallel command divides the cat output into blocks and dispatches it to the awk call, resulting in a number of sub-computation operations. These subcalculations go through the second pipeline into the same awk command, thus outputting the final result. * each awk has three backslashes, which is required for GNU parallel to call awk.

WC

Do you want to calculate the number of lines in a file as quickly as possible?

Traditional practice:

Wc-l bigfile.txt

Now you should go like this:

Cat bigfile.txt | parallel-- pipe wc-l | awk'{print s} 1} END {print s}'

Very ingenious, first use the parallel command 'mapping' to make a large number of wc-l calls to form a sub-calculation, and * send it to awk through the pipeline for summary.

SED

Do you want to do a lot of replacements with the sed command in a huge file?

General practice:

Sed s ^ new ^ g bigfile.txt

Now you can:

Cat bigfile.txt | parallel-- pipe sed s ^ new ^ g

... You can then use the pipe to store the output in the specified file.

Thank you for your reading, the above is the content of "how to accelerate the Linux command", after the study of this article, I believe you have a deeper understanding of how to accelerate the Linux command, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.