Shulou(Shulou.com)05/31 Report--

In order to solve the problem of how to migrate the data between COS, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and easy way.

The following describes Tencent Cloud inter-COS migration, and how to configure fully hosted public network migration tasks to achieve data migration.

Preparatory work

1. Tencent Cloud object Storage

1. Create a target storage space to hold the migrated data.

two。 Create a sub-user for migration and grant relevant permissions:

Log in to Tencent Cloud to access the console.

Select "user" > "user list" in the left navigation bar to go to the user list page.

For new sub-users, check programming access and Tencent Cloud console access.

Search and check QcloudCOSAccessForMSPRole and QcloudCOSFullAccess strategies.

Complete the sub-user to create and save the sub-user name, access login password, SecretId,SecretKey.

Note: the migration service can also be operated with the primary account, but for security reasons, it is recommended to create a new sub-user and use the sub-user API key for migration, and delete it after the migration is completed.

Operation steps

Log in to the migration service platform

1. Log in to the console to find the migration service platform, and click in the left navigation bar to go to the "Migration tool" page.

two。 Locate the "File Migration tool" module, and click "use now" to enter the file migration tool configuration page.

Create a new migration task

1. On the file migration tool page, click * * New Task * * to enter the new file migration task interface and set the migration parameters.

two。 Sets the migration task name.

Task name: the character length is 1 to 60 characters, and the allowed characters are Chinese, English, 0-9, _, -. The name set here will be used to view the migration status and progress in the task list.

3. Set the file source to be migrated.

Here, the migration source service provider should select Tencent Cloud COS, and enter the Tencent Cloud sub-user SecretId,SecretKey that was previously created for migration in the AccessKey,SecretKey text box below. The list of source object buckets can be obtained by clicking the refresh button to the right of the drop-down box after filling in the key.

4. Select the file storage method.

According to the needs of migration, set the storage mode of files after migration, you can choose: standard storage, low-frequency storage, keep the original storage attributes, archive storage.

5. Select the Header setting.

"if the files in the source bucket are set to Header/Tag and need to be retained after migration, select retain or set replacement rules."

6. Set migration rules.

Choose to migrate all files in the specified bucket or only files with the specified prefix.

7. Specifies when the migration task starts.

If you need to start the migration at the specified time, turn on this switch and set the start time.

8. Set the maximum number of concurrency.

The object storage of each public cloud vendor has the highest concurrency limit. To ensure business stability, confirm and set the maximum QPS available for migration with the source vendor before migration.

9. Select the destination to which you want to migrate.

In the migration target information, enter the Tencent Cloud sub-user SecretId,SecretKey for migration. The list of target object buckets can be obtained by clicking the refresh button on the right side of the drop-down box after filling in the key.

10. Specifies the specified directory to migrate to the target bucket.

For example:

The files in the source bucket are / a.txtMagneol Dirdy _ b.txt, and fill in the text box with "dest", then the path of the two files in the target bucket after migration is: / dest/a.txt,/dest/dir/b.txt.

If you fill in dest/20180901 in the text box, the path to the two files in the target bucket after migration is: / dest/20180901/a.txt,/dest/20180901/dir/b.txt.

Note: if the migration source and destination source have different contents and have the same name, it is recommended to select "Skip (keep the files with the same name in the target bucket)" in the "same name File" configuration. The system defaults to [overwrite (the file in the source bucket will overwrite the file with the same name in the target bucket)]. If the content of the file changes during the migration process, a second migration is required.

If the migration source and destination source have different contents and have the same name, it is recommended to select "Skip (keep the files with the same name in the target bucket)" in the "same name File" configuration, and the system defaults to [overwrite (the files in the source bucket will overwrite the files with the same name in the target bucket)].

If the content of the file changes during the migration process, a second migration is required.

Save to root: directly save the files in the source bucket to the root directory of the target bucket according to the original relative path.

Save to specified directory: saves the files in the source bucket to the specified directory with the original relative path.

11. Select the migration mode.

Start the fully managed migration immediately after the new migration task is created: select the managed migration. After the user clicks "New and start", the MSP service will access the source storage through the public network for migration.

Manually download Agent after creating a new migration task to start the migration: select Agent mode for migration. After the user clicks "New and start", only the task configuration will be created. Users need to manually download Agent to deploy on the server on the source side of the migration before the migration is officially started. Agent mode is suitable for scenarios where existing Direct Connect wants to migrate through Direct Connect.

twelve。 Click * * New and start * * to start the migration task.

III. Migration status and progress

In the main interface of the file migration tool, you can view the status and progress of all file migration tasks:

Task completed status, green indicates that the task is completed and all files are migrated successfully, and yellow indicates that the migration task is completed but some files fail to migrate.

After you click the retry failed task link, the failed files in the task will retry the migration, and the files that have been successfully migrated will not be retransmitted.

Click the Export link to export a list of files that failed during the migration.

IV. Migration completion time

The migration speed is determined by the minimum speed of each link involved in the migration process, and is also affected by the network transmission speed and the maximum number of concurrency.

The influencing factors are:

The influencing factors indicate that the reading speed of the moving source varies with different service providers, usually:

The transmission speed is between 50Mbps and 200Mbps.

File read concurrency is between 500-3000 (the transfer of a large number of small files is limited by concurrency).

The transfer speed of the MSP platform the MSP platform provides maximum 200Mbps migration bandwidth. Write speed of moving to the destination location: Tencent Cloud object storage COS: write transfer speed 200Mbps, write concurrency between 500-800.

The overall migration speed will be between 6MByte-25MByte (that is, 21GB/ hours-87GB/ hours). This is the answer to the question about how to transfer the data between COS. I hope the above content can be of some help to you. If you still have a lot of doubts to solve, you can follow the industry information channel for more related knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.