Shulou(Shulou.com)06/02 Report--

This article mainly introduces how to use multicore CPU to accelerate your Linux command, the article is very detailed, has a certain reference value, interested friends must read it!

Have you ever had the need to calculate a very large data (hundreds of GB)? Or search inside, or other operations-- something that can't be done in parallel. Data experts, I'm talking to you. You may have a 4-core or more-core CPU, but our appropriate tools, such as grep, bzip2, wc, awk, sed, etc., are single-threaded and can only use one CPU kernel.

To borrow the words of the cartoon character Cartman, "how can I use these cores"?

For the Linux command to use all the CPU kernels, we need to use the GNU Parallel command, which allows all of our CPU kernels to do magical map-reduce operations in a single machine, of course, with the rarely used-pipes parameter (also known as-spreadstdin). In this way, your load will be evenly distributed among the CPU, really.

BZIP2

Bzip2 is a better compression tool than gzip, but it's slow! Stop messing around, we have a way to solve the problem.

Previous practice:

Cat bigfile.bin | bzip2-- best > compressedfile.bz2

Now it goes like this:

Cat bigfile.bin | parallel-- pipe-- recend'--k bzip2-- best > compressedfile.bz2

Especially for bzip2,GNU parallel, it is super fast on multicore CPU. As soon as you are not careful, it will be finished.

GREP

If you have a very large text file, you might have done this before:

Grep pattern bigfile.txt

Now you can do this:

Cat bigfile.txt | parallel-- pipe grep 'pattern'

Or this:

Cat bigfile.txt | parallel-- block 10m-- pipe grep 'pattern'

This second use uses the-block 10m parameter, which means that each kernel processes 10 million rows-- you can use this parameter to adjust how many rows each CUP kernel processes.

AWK

Here is an example of using the awk command to calculate a very large data file.

General usage:

Cat rands20M.txt | awk'{print s} 1} END {slots}'

Now it goes like this:

Cat rands20M.txt | parallel-- pipe awk\'{slots =\ $1} END {print s}\'| awk'{print s} 1} END {slots}'

This is a bit complicated: the-pipe parameter in the parallel command divides the cat output into blocks and dispatches it to the awk call, resulting in a number of sub-computation operations. These subcalculations go through the second pipeline into the same awk command, thus outputting the final result. The first awk has three backslashes, which is required for GNU parallel to call awk.

WC

Do you want to calculate the number of lines in a file as quickly as possible?

Traditional practice:

Wc-l bigfile.txt

Now you should go like this:

Cat bigfile.txt | parallel-- pipe wc-l | awk'{print s} 1} END {print s}'

Very clever, first use the parallel command 'mapping' to make a large number of wc-l calls, form a sub-calculation, and finally send it to awk through the pipeline for summary.

SED

Do you want to do a lot of replacements with the sed command in a huge file?

General practice:

Sed s ^ new ^ g bigfile.txt

Now you can:

Cat bigfile.txt | parallel-- pipe sed s ^ new ^ g

... You can then use the pipe to store the output in the specified file.

The above is all the contents of the article "how to use multicore CPU to speed up your Linux commands". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.