Shulou(Shulou.com)05/31 Report--

This article introduces the knowledge of "how to use the presign of Multipart Upload". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Demand background

Some projects need to upload data on the client, but for security reasons, secret-key information cannot be stored anywhere on the client. At the same time, ordinary presign upload requests also have limitations, especially when the file is very large. Therefore, it is necessary to investigate whether Multipart Upload supports the presign mechanism. (it is not clear whether Multipart Upload supports presign,AWS or ceph.)

Basic process of Multipart Upload

The whole Multipart Upload is roughly divided into three stages.

1. Initiate_multipart_upload: use HTTP POST request to build initialization data. Metadata and content-type of object are all set at this stage. UploadID is returned in the response after the request is successful, and this uploadID is used to upload data later.

2. Upload_part: use HTTP PUT to request to build a data upload operation. The client splits large files into multiple part files locally. Each part uses the same uploadID, and uploads data in uploadNumber order (can be in parallel). Each part file uploaded successfully will return the corresponding etag to complete the consistency verification of each part file.

3. Compelte_upload: use the HTTP PUT request to build the final merge request operation to complete the logical merging of each part. Note that the returned etag does not have the consistency verification feature.

Summary of Multipart Upload features:

1. Split large files into multiple part files to achieve the upload of super-large files.

two。 Split part files can be uploaded in parallel, which greatly improves the upload efficiency.

3.Multipart Upload implements the process consistency check, not the final consistency check.

Information such as 4.metadata and content-type must be set up in advance during the initiate phase.

Implement presign in the upload_part phase

Process description: like ordinary Multipart Upload, split the files on the client side first, then generate a separate PreSignURL for each part file, and finally use the generated URL to upload each part file (no longer need secret-key).

Using python to implement presign in upload_part stage

It is divided into two parts. Server_demo.py mainly implements the most critical S3Sign. The code is as follows

#-*-coding: utf-8-*-import timeimport hmacfrom hashlib import sha1 as shapy3k = Falsetry: from urlparse import urlparse, unquote from base64 import encodestringexcept: py3k = True from urllib.parse import urlparse, unquote from base64 import encodebytes as encodestringclass S3PreSign (): def _ _ init__ (self, access_key, secret_key, service_url, bucket_name, object_name, upload_ID Expires): self.service_url = str (service_url) self.access_key = str (access_key) self.secret_key = str (secret_key) self.bucket_name = str (bucket_name) self.object_name = str (object_name) self.upload_ID = str (upload_ID) self.Expires = int (time.time ()) + int (expires) def get_signature_str (self Sign_str): if py3k: key = self.secret_key.encode ('utf-8') msg = sign_str.encode (' utf-8') else: key = self.secret_key msg = sign_str h = hmac.new (key, msg, digestmod=sha) return (encodestring (h.digest ()) .strip ()) .replace ('+' '% 2b') def build_url (self, partNumber, Signature): url_ = "http://{bucket_name}.{service_url}/{object_name}?uploadId={uploadId}&partNumber={partNumber}&Expires={Expires}&AWSAccessKeyId={AWSAccessKeyId}&Signature={Signature}".format( bucket_name=self.bucket_name, service_url=self.service_url, object_name=self.object_name, uploadId=self.upload_ID PartNumber=partNumber, Expires=self.Expires, AWSAccessKeyId=self.access_key, Signature=Signature) return url_ def build_url_with_partid (self, partMd5, partNumber): sign_str = "PUT\ n {partMd5}\ n\ n {Expires}\ n / {bucket_name} / {object_name}? partNumber= {partNumber} & uploadId= {uploadId}" .format (partMd5=partMd5 Expires=self.Expires, bucket_name=self.bucket_name, object_name=self.object_name, partNumber=partNumber, uploadId=self.upload_ID) Signature_ = self.get_signature_str (sign_str) return self.build_url (partNumber, Signature_)

Finally, the corresponding parallel upload use case is implemented in client_demo.py. Here is just a simple description of the process. If you need to hide the information of secret-key, you need to package the process of generating presignURL into an independent service interface, and the client can obtain the generated presignURL by request.

#-*-coding: utf-8-*-from server_demo import S3PreSignimport requestsfrom base64 import encodestringfrom hashlib import md5import osfrom multiprocessing import Poolclass S3client (): def _ _ init__ (self, part_num Uploadfile_path): self.part_num = part_num self.uploadfile_path = uploadfile_path def split_file (self): filelist = [] statinfo = os.stat (self.uploadfile_path) chunksize = statinfo.st_size / self.part_num print "File size:% d (MB)"% (statinfo.st_size / (1024 * 1024)) print self.uploadfile_path Chunksize with open (self.uploadfile_path, "rb") as f: index = 1 while True: chunk = f.read (chunksize) if (chunk): fn = "% s.part.%d"% (self.uploadfile_path, index) print "creating", fn with open (fn) "wb") as fw: fw.write (chunk) partMD5 = self.compute_hash (fn) tmp_ = {} tmp_ [fn] = str (partMD5) filelist.append (tmp_) index = index + 1 Else: break return filelist def compute_hash (self Filepath, buf_size=8192, size=None, hash_algorithm=md5): hash_obj = hash_algorithm () with open (filepath) as fp: spos = fp.tell () if size and size < buf_size: s = fp.read (size) else: s = fp.read (buf_size) while s: if not isinstance (s Bytes): s = s.encode ('utf-8') hash_obj.update (s) if size: size-= len (s) if size

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.