Shulou(Shulou.com)05/31 Report--

This article mainly introduces the RGW S3 Multipart example analysis, has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let the editor take you to understand it.

S3 segmented upload technology is mainly used in the data upload of large files, usually in the S3 client will do a slicing operation on the uploaded large files. Within the RGW, the data sent by the S3 client will be shredded again. The default shard size for RGW is 4MB.

1. Initialize Multipart.

The HTTP POST operation is used to initialize Multipart, and the RGWInitMultipart_ObjStore_S3 class processing is used in RGW. Because the RGWInitMultipart_ObjStore_S3 class inherits from RGWInitMultipart_ObjStore, the RGWInitMultipart_ObjStore class inherits from RGWInitMultipart, and the RGWInitMultipart class inherits from the RGWOp class. Therefore, the operations that deal with Mulipart initialization are mainly concentrated in the RGWInitMultipart::execute () function. The processing flow of this function is parsed in detail below.

RGWInitMultipart::execute ()

| | _ _ RGWInitMultipart_ObjStore_S3::get_params () gets the RGWAccessControlPolicy_S3 class instance from the HTTP request |

| | _ _ put RGWAccessControlPolicy_S3 into the attribute list and attribute key=RGW_ATTR_ACL |

| _ _ rgw_get_request_metadata () gets the attribute information from the HTTP request and the prefix key of these attribute information is user.rgw.

| _ _ generate upload_id. Upload_id = "2 ~" + 32 random alphanumeric characters

| | _ _ initialize RGWMPObj object with object name and upload_id as parameters |

| | _ _ initializes the rgw_obj object with bucket, RGWMPObj.meta and RGWMPObj.ns as parameters |

| | _ _ create RGWRados::Object object |

| | _ _ create RGWRados::Object::Write object |

| | _ RGWRados::Object::Write::write_meta () |

| | _ _ create ObjectWriteOperation object |

| | _ _ write the key/value of attribute attrs to the ObjectWriteOperation object |

| | _ _ create RGWRados::Bucket object |

| | _ _ create RGWRados::Bucket::UpdateIndex object |

| | _ _ RGWRados::Bucket::UpdateIndex::prepare (CLS_RGW_OP_ADD) prepares to update the header information and dir_entry information of the bucket where the object resides |

| | _ RGWRados::cls_obj_prepare_op () |

| | _ cls_rgw_bucket_prepare_op () |

| | _ rgw_bucket_prepare_op () |

| | _ _ read_key_entry (). For entry that does not exist, create an instance of rgw_bucket_dir_entry class |

| | _ _ read_bucket_header () reads rgw_bucket_dir_header |

| | _ _ cls_cxx_map_set_val () writes rgw_bucket_dir_entry to ceph cluster |

| | _ _ write_bucket_header () writes updated rgw_bucket_dir_header information to the ceph cluster |

| | _ _ RGWRados::IoCtx::operate () performs the operation of writing the attrs of object to the ceph cluster |

| | _ RGWRados::Bucket::UpdateIndex::complete () |

| | _ RGWRados::cls_obj_complete_add () |

| | _ RGWRados::cls_obj_complete_op () |

| | _ cls_rgw_bucket_complete_op () |

| | _ rgw_bucket_complete_op () |

| | _ _ read_bucket_header () reads rgw_bucket_dir_header |

| | _ _ read_key_entry (). Create an instance of rgw_bucket_dir_entry class for entry that does not exist |

| | _ _ cls_cxx_map_set_val () writes rgw_bucket_dir_entry to ceph cluster |

| | _ _ write_bucket_header () writes updated rgw_bucket_dir_header information to the ceph cluster |

2. Upload Multipart data.

The HTTP PUT operation is used to upload Multipart data, and the RGWPutObj_ObjStore_S3 class is used in RGW. Because the RGWPutObj_ObjStore_S3 class inherits from RGWPutObj_ObjStore, the RGWPutObj_ObjStore class inherits from RGWPutObj, and the RGWPutObj class inherits from the RGWOp class. Therefore, the operations that deal with Mulipart initialization are mainly concentrated in the RGWPutObj::execute () function. The RGWPutObjProcessor_Multipart class is mainly responsible for handling the Multipart part of this function. There are two main handling functions in this class, prepare () and do_complete (). The following parses the processing flow of these two functions respectively.

RGWPutObjProcessor_Multipart::prepare ()

| | _ _ get uploadId and partNumber information from HTTP request flow |

| | _ _ set manifest.prefix to oid+ "." + uploadId |

| | _ RGWObjManifest::set_multipart_part_rule () |

| | _ RGWObjManifest::generator::create_begin () |

| | _ _ RGWObjManifest::get_implicit_location () gets the name and location of Multipart data |

| | _ RGWObjManifest::update_iterators () |

RGWPutObjProcessor_Multipart::do_complete ()

| | _ _ create RGWRados::Object object |

| | _ _ create RGWRados::Object::Write object |

| | _ RGWRados::Object::Write::write_meta () |

| | _ _ RGWRados::omap_set () updates the omap information of the multipart_meta_obj object |

3. Upload Multipart data.

The HTTP POST operation is used to complete the Multipart, and the RGWCompleteMultipart_ObjStore_S3 class processing is used in RGW. Because the RGWCompleteMultipart_ObjStore_S3 class inherits from RGWCompleteMultipart_ObjStore, the RGWCompleteMultipart_ObjStore class inherits from RGWCompleteMultipart, and the RGWCompleteMultipart class inherits from the RGWOp class. Therefore, the operation done by processing Mulipart is mainly concentrated in the RGWCompleteMultipart::execute () function. The processing flow of this function is parsed in detail below.

RGWCompleteMultipart::execute ()

| | _ _ get_obj_attrs () gets the attribute value of the Multipart object |

| | _ _ list_multipart_parts () traverses all the shards of Multipart object |

| | _ _ parses all shards and writes the manifest information of the shards to the manifest list of Multipart objects |

| | _ _ create RGWRados::Object object |

| | _ _ create RGWRados::Object::Write object |

| | _ _ RGWRados::Object::Write::write_meta () writes attribute values to the Ceph cluster |

| | _ _ RGWRados::delete_obj () Delete meta_obj object (this object is responsible for saving object information during multipart upload) |

4. Terminate Multipart data upload.

The HTTP DELETE operation is used to terminate the Multipart, and the RGWAbortMultipart_ObjStore_S3 class processing is used in RGW. Because the RGWAbortMultipart_ObjStore_S3 class inherits from RGWAbortMultipart_ObjStore, the RGWAbortMultipart_ObjStore class inherits from RGWAbortMultipart, and the RGWAbortMultipart class inherits from the RGWOp class. Therefore, the processing of Mulipart termination is mainly concentrated in the RGWAbortMultipart::execute () function. The processing flow of this function is parsed in detail below.

RGWAbortMultipart::execute ()

| | _ _ get_multipart_info () gets the attribute value of the meta_oid object |

| | _ _ list_multipart_parts () traverses all the shards of Multipart object |

| | _ _ RGWRados::update_gc_chain () puts the completed Multipart part into the GC queue and waits for the GC thread to do the recycling operation |

| | _ _ RGWRados::send_chain_to_gc () starts GC thread |

| | _ _ create RGWRados::Object object |

| | _ _ create RGWRados::Object::Delete object |

| | _ _ RGWRados::Object::Delete::delete_obj () deletes meta_oid object |

5. View Multipart data.

See that Multipart uses the HTTP GET operation, which is handled by the RGWListMultipart_ObjStore_S3 class in RGW. Because the RGWListMultipart_ObjStore_S3 class inherits from RGWListMultipart_ObjStore, the RGWListMultipart_ObjStore class inherits from RGWListMultipart, and the RGWListMultipart class inherits from the RGWOp class. Therefore, the processing of Mulipart termination is mainly concentrated in the RGWListMultipart::execute () function. The processing flow of this function is parsed in detail below.

RGWListMultipart::execute ()

| | _ _ get_multipart_info () gets the attribute value of the meta_oid object |

| | _ _ list_multipart_parts () traverses all the shards of Multipart object |

Here is the class diagram for RGW S3 Multipart

Thank you for reading this article carefully. I hope the article "sample Analysis of RGW S3 Multipart" shared by the editor will be helpful to you. At the same time, I also hope you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.