Shulou(Shulou.com)05/31 Report--

This article introduces the relevant knowledge of "how to realize s3cmd data operation". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

1. Basic principles

Operation procedure:

1. The client finishes slicing the file, and then submits the upload operation request to API server to generate the corresponding presign URL (steps 1 and 2. If you want to control the number of client uploads, you can generate a specified number of token at this stage)

two。 Use the generated presign URL to construct a HTTP request to upload data to the S3 service. (step 3)

3. After the client completes all multipart uploads, it submits the Complete request to API server, and then API server sends the complete request to the S3 service (steps 4 and 5)

4. The client gets the return of API server and completes the final operation. (you can consider recycling token here)

two。 Advantages and disadvantages

Advantages:

1. Accesskey and secretkey will not be stored on the client side to avoid key leakage

two。 Each presignURL corresponds to a keyname, which can upload and overwrite existing files at will within the valid time, which is more flexible.

3. The server can combine various Auth systems to complete the authentication and authorization of the client, and it is convenient to integrate existing services.

4. The client upload and download method is flexible. After getting the presignURL, you can upload and download through any client that supports the HTTP protocol.

5. It is suitable for uploading large files. Compared with the Presign method introduced in this paper, concurrent upload is supported in the data upload phase, and the upload efficiency is greatly improved.

Disadvantages:

1. The upload step requires multiple interactions, and the process is a little more complicated.

two。 Limited S3 multipart upload standard, this method is not suitable for files less than 5m.

3. Concrete realization

Install server-side dependencies

Pip install botopip install flask-restful

The server demo code is as follows:

#-*-coding: utf-8-*-import timeimport hmacfrom hashlib import sha1 as shaimport botoimport boto.s3.connectionimport repy3k = Falsetry: from urlparse import urlparse, unquote from base64 import encodestringexcept: py3k = True from urllib.parse import urlparse, unquote from base64 import encodebytes as encodestringfrom flask import Flask, requestfrom flask_restful import Api, Resourceapp = Flask (_ _ name__) api = Api (app) from boto.s3.multipart import MultiPartUploadclass MultiPartUpload_Presign (MultiPartUpload): def _ init__ (self,id,bucket Key_name): MultiPartUpload.__init__ (self) self.id = id self.bucket = bucket self.key_name = key_name def complete_upload (self): xml = self.to_xml () return self.bucket.complete_multipart_upload (self.key_name, self.id Xml) class S3PreSign (): def _ _ init__ (self, object_name,metadata=None) Policy=None): self.service_url = 's3.ceph.work' # fill in the S3 service endpoint self.access_key =''# access key self.secret_key =''# secret key self.bucket_name = 'multi-upload' # bucket name self.object_name = str (object_name) # self.Expires = int (time.time ()) + int (expires) conn = boto.connect_s3 (aws_access_key_id = self.access_key Aws_secret_access_key = self.secret_key, host = self.service_url, port = 80, is_secure=False, # uncommmnt if you are not using ssl # calling_format = boto.s3.connection.OrdinaryCallingFormat (), calling_format = boto.s3.connection.SubdomainCallingFormat () ) self.bucket = conn.get_bucket (self.bucket_name) self.upload_ID = self.Make_uploadID (self.object_name, metadata=metadata, policy=policy) def Make_uploadID (self,object_name,metadata=None,policy=None): mpu = self.bucket.initiate_multipart_upload (object_name,metadata= metadata, policy=policy) return mpu.id def complete_upload (self,upload_ID): mpu = MultiPartUpload_Presign (id=upload_ID, bucket=self.bucket) Key_name=self.object_name) status_ = 200try: mpu.complete_upload () except: status_ = 422finally: return status_ def get_signature_str (self Sign_str): if py3k: key = self.secret_key.encode ('utf-8') msg = sign_str.encode (' utf-8') else: key = self.secret_key msg = sign_str h = hmac.new (key, msg, digestmod=sha) return (encodestring (h.digest ()) .strip ()) .replace ('+' '% 2b') def build_url (self, expires,partNumber, Signature): url_ = "http://{bucket_name}.{service_url}/{object_name}?uploadId={uploadId}&partNumber={partNumber}&Expires={Expires}&AWSAccessKeyId={AWSAccessKeyId}&Signature={Signature}".format( bucket_name=self.bucket_name, service_url=self.service_url, object_name=self.object_name UploadId=self.upload_ID, partNumber=partNumber, Expires= expires, AWSAccessKeyId=self.access_key, Signature=Signature) return url_ def build_url_with_partid (self,expires, partNumber) PartMd5): sign_str = "PUT\ n {partMd5}\ n\ n {Expires} / {bucket_name} / {object_name}? partNumber= {partNumber} & uploadId= {uploadId}" .format (partMd5=partMd5, Expires=expires, bucket_name=self.bucket_name, object_name=self.object_name, partNumber=partNumber UploadId=self.upload_ID) Signature_ = self.get_signature_str (sign_str) return self.build_url (expires, partNumber) Signature_) class MultiPart_List (Resource): def post (self): PartNumber_ = {} metadata = {} policy = None # print request.form ['keyname'] if' keyname' in request.form: keyname = request.form ['keyname'] else: return "no key" 400 if 'expires' in request.form: expires = request.form [' expires'] else: return "no expires" 400 if 'contenttype' in request.form: metadata [' Content-Type'] = str (request.form ['contenttype']) if 'xMuramzMuthacl`in request.form: policy = str (request.form [' xMuramzMuacl`]) for part_ in request.form: if re.match (r'^\ d {1,} $') Part_): # print part_ PartNumber_ [part_] = request.form [part_] meatadata_rule = 'xMuamizmaimei' if re.match (meatadata_rule, part_): # print part_ metadata [part _ .split (meatadata_rule) [1]] = str (request. Form [part _]) print metadata,policy,keyname Expires s3client = S3PreSign (keyname) result = {} result ['UploadID'] = s3client.upload_ID expires = int (time.time ()) + int (expires) for p _ in PartNumber_: result [p] = s3client.build_url_with_partid (expires,p_,PartNumber_ [p]) return result 201class Complete_MultiPart (Resource): def post (self): if 'keyname' in request.form: keyname = request.form [' keyname'] else: return "no key", 400 if 'uploadid' in request.form: uploadid = request.form [' uploadid'] else: return "no UploadID" 400 s3client = S3PreSign (keyname) result = s3client.complete_upload (uploadid) return {"status_code": result}, resultapi.add_resource (MultiPart_List,'/ presign') api.add_resource (Complete_MultiPart,'/ complete') if _ _ name__ = ='_ main__': app.run (debug=True)

Install client dependency

Pip install requests

The client demo code is as follows:

#-*-coding: utf-8-*-import requestsfrom base64 import encodestringfrom hashlib import md5import osimport jsonfrom multiprocessing import Pooldef multipart_upload_with_part (url_, part_file_path, partMD5): headers= {} headers ["Content-MD5"] = partMD5 with open (part_file_path,'r') as fh: response = requests.put (url_, headers=headers) Data=fh.read () if response.status_code = = 200: print "{} upload Sucessful!" .format (part_file_path) class S3client (): def _ _ init__ (self, key_name, expires,part_num, uploadfile_path, policy=None, contenttype=None, metadata=None Processes_num=2): self.multipart_data = {} if key_name: self.multipart_data ['keyname'] = key_name if expires: self.multipart_data [' expires'] = expires if policy: self.multipart_data ['xmuramzMuthacl'] = policy if contenttype: self.multipart_data [' contenttype'] = contenttype if metadata: for k in metadata: self.multipart_ data [k] = metadata [k] self.part_num = part_num self.processes_num = processes_num self.uploadfile_path = uploadfile_path self.server = 'http://localhost:5000/' # here fill in your API server address self.upload_file_list_ = { } def split_file (self): filelist = [] statinfo = os.stat (self.uploadfile_path) chunksize = statinfo.st_size / self.part_num print "File size:% d (MB)"% (statinfo.st_size / (1024 * 1024)) print self.uploadfile_path Chunksize with open (self.uploadfile_path, "rb") as f: index = 1 while True: chunk = f.read (chunksize) if (chunk): fn = "% s.part.%d"% (self.uploadfile_path, index) # print "creating", fn with open (fn "wb") as fw: fw.write (chunk) partMD5 = self.compute_hash (fn) tmp_ = {} tmp_ [fn] = str (partMD5) filelist.append (tmp_) index = index + 1 Else: break return filelist def compute_hash (self Filepath, buf_size=8192, size=None, hash_algorithm=md5): hash_obj = hash_algorithm () with open (filepath) as fp: spos = fp.tell () if size and size < buf_size: s = fp.read (size) else: s = fp.read (buf_size) while s: if not isinstance (s Bytes): s = s.encode ('utf-8') hash_obj.update (s) if size: size-= len (s) if size

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.