Shulou(Shulou.com)05/31 Report--

Today, I will talk to you about the example analysis of ossutil upload performance tuning, which may not be well understood by many people. in order to make you understand better, the editor has summarized the following for you. I hope you can get something according to this article.

Abstract: ossutil is often asked by internal students or external customers about concurrent upload performance. This paper briefly describes the principle of ossutil concurrent upload and gives an example. Users can obtain ossutil from here. Parameter-when recursive uploads files to oss, if file_url is a directory, you must specify the-- recursive option, otherwise you do not need to specify the-- recursive option.

Often encounter internal students or external customers ask ossutil about concurrent upload performance. This paper briefly describes the principle of ossutil concurrent upload and gives an example.

Users can obtain ossutil from here.

Parameter-recursive

When uploading files to oss, if file_url is a directory, you must specify the-- recursive option, otherwise you do not need to specify the-- recursive option.

When downloading files from oss or copying files between oss

If the-- recursive option is not specified, a single object is considered to be copied. At this point, make sure that the src_url specifies exactly the object to be copied, and if the object does not exist, an error is reported.

If the-- recursive option is specified, ossutil performs a prefix match lookup on the src_url, and for these objects bulk copies, if the copy fails, the copy that has been performed will not be rolled back.

When uploading (or downloading or copying) files in batches, if one of the file operations fails, ossutil will not quit, but will continue to upload (or download, copy) other files, and record the error information of the error files in the report file. File information that was successfully uploaded (or downloaded or copied) will not be recorded in the report file.

Termination of operation when an error occurs in a batch operation

If you do not enter the batch file iteration process and the error has occurred, the report file will not be generated and the ossutil will stop running. For example, when a user enters an error in the cp command, the report file is not generated, but the screen output error and exit.

If an error occurs in a file during the batch operation: Bucket does not exist, illegal permission verification caused by accessKeyID/accessKeySecret error, ossutil will output the error on the screen and exit.

The report file is called: ossutil_report_ date _ time. Report. The report file is a kind of ossutil output file, which is placed under the output directory of ossutil. The path to this directory can be specified with the outputDir option in the configuration file or the command line-- output-dir option. If not specified, the default output directory is used: the ossutil_output directory under the current directory.

Ossutil does not do the maintenance of report files, please check and clean up users' report files to avoid generating too many report files.

Concurrency control parameter

-- the jobs option controls the number of concurrency initiated between files when multiple files are uploaded / downloaded / copied

-- parallel controls the number of concurrency between fragments when uploading / downloading / copying large files.

By default, ossutil calculates the number of parallel based on file size (this option does not work for small files, and the threshold for multipart upload / download / copy of large files can be controlled by the-- bigfile-threshold option). When uploading / downloading / copying large files in bulk, the actual concurrency is the number of jobs multiplied by the number of parallel. These two options can be adjusted by the user, and when the default set by ossutil is less than the user's performance needs, the user can adjust the two options to improve or decrease the performance.

-- part-size option

This option sets the size of each part when uploading / downloading / copying large files in parts.

By default, you do not need to set this value. Ossutil determines the shard size and sharding concurrency according to the file size. You can set these options when the upload / download / copy performance does not meet the requirements, or when you have other special needs.

If this option (shard size) is set, the number of shards is: rounded up (file size / shard size). Note that if the value of the-- parallel option is greater than the number of shards, then the excess parallel will not work, and the actual number of concurrency is the number of shards.

If you set the part size value too low, it may affect the performance of uploading / downloading / copying ossutil files. If you set it too high, it will affect the actual number of multipart concurrency that works, so please set the part size option value reasonably.

Performance tuning

If the concurrency is adjusted too much, the upload / download / copy performance of ossutil may decline due to resource switching and snatching between threads, so adjust the values of these two options according to the actual machine conditions. If you want to carry out the stress test, you can lower the two values at the beginning and slowly increase to find the optimal value.

If the values of the-- jobs option and-- parallel option are too large, the EOF error may occur because the network transmission is too slow in the case of limited machine resources. Please reduce the-jobs option and-- parallel option value appropriately.

If the number of files is too many and the size is uneven, use-- jobs=3-- parallel=4 directly at the same time (the concurrency between files is 3, and the concurrency within a single file is 4). At the same time, observe the MEM, CPU, and network conditions. If the network is not full and CPU is full, you can continue to raise-- jobs and-- parallel.

Real case

Cdn.com/d05c9a5264819e0ef373602d64f5392f1d0e462d.png ">

According to the customer scenario at that time, the download speed was about 265M/s.

Case analysis

By default, because it is a multi-file download, five files (version) are downloaded at the same time

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.