Shulou(Shulou.com)06/01 Report--

This article mainly introduces the relevant knowledge of "how to use CPU to accelerate the execution speed of Linux commands". The editor shows you the operation process through an actual case. The operation method is simple, fast and practical. I hope this article "how to use CPU to accelerate the execution speed of Linux commands" can help you solve the problem.

When dealing with big data, we always think of some parallel operations to speed up our operations. our cpu is multi-core and multi-threaded, but some of our commands are single-threaded commands that cannot perform parallel operations, such as grep, bzip2, wc, awk, sed and so on. We can only use one CPU kernel. For the Linux command to use all the CPU kernels, we need to use the GNU Parallel command.

BZIP2

Bzip2 is a better compression tool than gzip, but it's slow! Stop messing around, we have a way to solve the problem.

Previous practice:

Cat bigfile.bin | bzip2-- best > compressedfile.bz2

Now it goes like this:

Cat bigfile.bin | parallel-- pipe-- recend'--k bzip2-- best > compressedfile.bz2

Especially for bzip2,GNU parallel, it is super fast on multicore CPU. As soon as you are not careful, it will be finished.

GREP

If you have a very large text file, you might have done this before:

Grep pattern bigfile.txt

Now you can do this:

Cat bigfile.txt | parallel-- pipe grep 'pattern'

Or this:

Cat bigfile.txt | parallel-- block 10m-- pipe grep 'pattern'

This second use uses the block 10m parameter, which means that each kernel processes 10 million rows, which you can use to adjust how many rows each CUP kernel processes.

AWK

Here is an example of using the awk command to calculate a very large data file.

General usage:

Cat rands20M.txt | awk'{print s} 1} END {slots}'

Now it goes like this:

Cat rands20M.txt | parallel-- pipe awk\'{slots =\ $1} END {print s}\'| awk'{print s} 1} END {slots}'

This is a bit complicated: the-pipe parameter in the parallel command divides the cat output into blocks and dispatches it to the awk call, resulting in a number of sub-computation operations. These subcalculations go through the second pipeline into the same awk command, thus outputting the final result. The first awk has three backslashes, which is required for GNU parallel to call awk.

WC

Do you want to calculate the number of lines in a file as quickly as possible?

Traditional practice:

Wc-l bigfile.txt

Now you should go like this:

Cat bigfile.txt | parallel-- pipe wc-l | awk'{print s} 1} END {print s}'

Very clever, first use the parallel command mapping to make a large number of wc-l calls, form a sub-calculation, and finally send it to awk through the pipeline for summary.

SED

Do you want to do a lot of replacements with the sed command in a huge file?

General practice:

Sed s ^ new ^ g bigfile.txt

Now you can:

Cat bigfile.txt | parallel-- pipe sed s ^ new ^ g

You can then use the pipe to store the output in the specified file.

That's all for "how to use CPU to speed up the execution of Linux commands". Thank you for reading. If you want to know more about the industry, you can follow the industry information channel. The editor will update different knowledge points for you every day.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.