Shulou(Shulou.com)06/01 Report--

What is the solution to speed up gzip compression? I believe many inexperienced people are at a loss about it. Therefore, this article summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

What is pig?

The code is as follows:

A parallel implementation of gzip for modernmulti-processor, multi-core machines .

To put it simply, it supports parallel gzip. Let's cut the crap and start the test.

By default, pig uses the current number of logical cpu to concurrently compress. If the number cannot be detected, 8 threads will be concurrently.

1. Test package file 1

The code is as follows:

Before compression

-rw-r--r-- 1 root root 2474772480 Dec 3 19:54 dbs.tar

1. Gzip test

Compression takes time, and CPU is the highest: 101.6%

Real 2m18.042s

User 2m10.204s

Sys 0m6.876s

Decompression takes time, and CPU is the highest: 101.6%.

Real 0m26.756s

User 0m20.175s

Sys 0m6.580s

After compression

-rw-r--r-- 1 root root 2349563863 Dec 3 19:54 dbs.tar.gz

2. Pigz test

Concurrent 8 threads (default):

Compression takes time, and CPU is the highest: 832%

Real 0m21.710s

User 2m31.229s

Sys 0m9.101s

Decompression takes time, and CPU is the highest: 199%.

Real 0m13.239s

User 0m11.791s

Sys 0m12.426s

Concurrent 4 threads:

Compression takes time, and CPU is the highest: 427.6%

Real 0m38.299s

User 2m30.701s

Sys 0m8.369s

Decompression takes time. CPU is the highest: 202%.

Real 0m13.294s

User 0m12.154s

Sys 0m11.862s

After compression

-rw-r--r-- 1 root root 2350161547 Dec 3 19:54 dbs.tar.gz

II. Test package file 2

Before compression

-rw-r--r-- 1 root root 5248890880 Dec 3 20:26 mysql-5.5.28-linux2.6-x86_64.tar

1. Gzip test

Compression time-consuming, with the highest CPU: 101%

Real 6m20.524s

User 6m2.466s

Sys 0m7.410s

Decompression takes time, and CPU is the highest: 101%

Real 0m58.893s

User 0m46.258s

Sys 0m11.944s

After compression

-rw-r--r-- 1 root root 1410045513 Dec 3 20:26 mysql-5.5.28-linux2.6-x86_64.tar.gz

2. Pigz test

Concurrent 8 threads (default):

Compression takes time, and CPU is the highest: 799%

Real 1m11.908s

User 6m36.717s

Sys 0m9.860s

Decompression takes time, and CPU is the highest: 179%

Real 0m33.188s

User 0m38.069s

Sys 0m16.728s

Concurrent 4 threads:

Compression takes time, and CPU is the highest: 418%

Real 1m41.547s

User 6m36.782s

Sys 0m9.806s

Decompression takes time, and CPU is the highest: 179%

Real 0m33.120s

User 0m37.500s

Sys 0m16.956s

After compression

-rw-r--r-- 1 root root 1406061442 Dec 3 20:26 mysql-5.5.28-linux2.6-x86_64.tar.gz

1. The default usage of pigz (the default number of concurrent threads is the number of logical cpu) is 5.3 times faster than gzip, the consumption of CPU is 8 times that of gzip, and the compression ratio is equivalent.

2. Concurrent 8 threads vs. 4 threads: 41.2% focus 16 threads vs. 8 threads: 27.9% dint 32 threads vs. 16 threads: 3%

3. In scenarios that require high compression efficiency, but high CPU consumption in a short period of time, pigz is very suitable.

After reading the above, have you mastered the solution to speed up gzip compression? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.