Shulou(Shulou.com)05/31 Report--

This article mainly introduces the relevant knowledge of how to use image-transfer, the content is detailed and easy to understand, the operation is simple and fast, and has a certain reference value. I believe you will gain something after reading this article on how to use image-transfer. Let's take a look at it.

How to migrate images to the cloud in business?

There are mainly several cloud scenarios in business, one is self-built idc cloud, the second is third-party cloud migration, and the third is remote disaster recovery, namely hybrid cloud. In these scenarios, whether it is cloud migration or cloud migration, the migration process mainly includes the following steps.

Network planning. Vpc configuration, subnetting, etc.

Data migration. Object storage, file storage and so on.

Application migration. Business, configuration, etc.

In the process of data migration, if the business uses containerized deployment heavily, batch image migration is required. At present, the image warehouse services of most cloud vendors do not provide the ability to migrate image repositories in batch. If the business wants to migrate, it can only download the original image locally, modify the tag, and then upload it to the destination image repository. There are several shortcomings in this process:

It takes time and effort. For a single image migration alone, three steps are required, and you need to keep an eye on it all the time. If you fail, you need to deal with it further.

Complete human flesh operation, prone to errors. When modifying the tag of the original image, it is easy to make mistakes due to artificial modification, and the tag is modified incorrectly, etc.

When the amount of mirror image is large, the cost increases and the progress on cloud is slow. If there are hundreds of images, such a human flesh operation, the migration progress will be very slow, and the migration cost will be greatly increased.

The principle and design goal of image-transfer

Pain points for mirror migration. We want to develop a batch migration tool for images. For users, batch migration of images can be achieved with simple configuration, without human intervention, to improve the progress of cloud / migration in business and reduce costs. Specifically, the tool has the following design goals.

The configuration is simple and no complex input is needed. We hope that this tool is easy to use for users, and batch migration can be completed by configuring source, destination image address and authentication.

Migrate a large number of images quickly and efficiently to reduce the migration cost. In view of a large number of image migration requirements, we hope that the tools can be completed efficiently, with the help of some technologies with high concurrency to achieve rapid migration.

Certain fault tolerance, reduce manual intervention and maintenance. In the process of migration, we hope that the tool can carry out some error identification and automatic repair, minimize manual maintenance and improve the efficiency of operation and maintenance.

No dependence on the running environment to improve the universality of tools. We want the tool to run in any linux,mac os,windows operating system without relying on docker and other programs.

Tencent Cloud CCR one-click migration mode is supported. Currently, Tencent Cloud container image service TCR Enterprise Edition is officially launched, and Tencent Cloud TCR personal Edition (CCR) will gradually reduce maintenance until it goes offline. Therefore, the tool needs to support full migration of CCR repositories to TCR Enterprise Edition with one click.

Custom qps speed limit is supported. When images are migrated in batches, frequent calls to the image repository API may cause the image repository to crash, so qps needs to be restricted.

Architecture

Image-transfer consists of several modules, and the following figure shows the architecture of image-transfer.

Common mode input (default): used to accept image migration tasks issued by users. Includes image migration profile and authentication profile. This model is used to realize image migration between cloud vendors.

CCR one-click migration mode input: you need to add-- ccrToTcr=true in the tool input parameter. This mode is used for CCR warehouse full-click migration to TCR Enterprise Edition. In addition to adding the-- ccrToTcr=true parameter, authentication configuration file and Tencent Cloud secret configuration file are also required.

Pipeline: this module is at the core of the tool's handling of image migration. It is responsible for handling the image migration tasks issued by users, including the synchronization rules of the image repository according to the migration configuration file, as well as the hierarchical pull and transfer tasks of the image. The module adopts the pipeline model with high concurrency to improve the migration speed.

Retry task: this phase retries tasks in pipeline that failed to transfer. The number of retries can be determined according to the user input parameters, and the default is 2.

Pipeline

Because the tool is written in golang language, the core processing module adopts go's pipeline high concurrency model. The whole pipeline module is divided into three small modules.

Synchronization rules of Multi-process processing Mirror Warehouse

Here, the image migration configuration file entered by the user is processed and parsed to obtain the source image address (including repo and tag) and the corresponding destination image address that need to be transferred. Then a job is formed for each source image address and destination image address. If the tag of the source image is not specified in the image configuration file, all the tag under the repo will be pulled, and a job will be formed for each tag. This process is handled by the multi-co-programming mode of golang to increase the processing speed. The number of collaborators can be specified by the user in the input parameter-- proc. The default is 5. After each job is composed, it is put into the task channel, waiting to be consumed.

Task transmission channel

The task channel can be seen as a simple middleware implemented by golang's channel. After each job is produced, it will be put into the channel, waiting to be consumed. The design can ensure that the job produced by the producer will be put into the consumption line immediately, and once the consumer has free time, it can be processed for consumption. Improve the efficiency of job processing.

Multi-process task

These collaborations are the consumer side of job. After getting the job, you will first pull the manifest of the source address in the job to determine whether it is a multi-manifest image, then pull each blob, then transfer the blob to the destination address, and finally transfer the manifest to the destination address. The whole process is to use the cache to keep the data from disk to improve efficiency. This process is handled by the multi-co-programming mode of golang to increase the processing speed. The number of collaborators can be specified by the user in the input parameter-- routines. The default is 5.

Best practices for Image Migration

This section describes how to use image-transfer tools to achieve batch image migration in different scenarios. The scenarios are as follows:

Image migration between different cloud vendors. For example, migrate from Aliyun Image Repository ACR to Tencent Cloud Image Warehouse TCR.

Open source / self-built image warehouse is migrated to the cloud. For example, migrate from harbor image warehouse to Tencent Cloud image warehouse TCR.

Tencent Cloud TCR personal Edition (CCR) migrates to Tencent Cloud Image Warehouse Enterprise Edition TCR with one click.

Tool installation:

First of all, there are two ways to download and compile the tool, one is to get the binary file directly, and the second is to download the source code to compile.

Download address of binary release package:

Https://github.com/tkestack/image-transfer/releases

The example is demonstrated by downloading source code compilation:

The compilation process is also very simple, go to the source directory and make directly.

Git clone https://github.com/tkestack/image-transfer.gitcd. / image-transfermake

When the compilation is complete, the image-transfer binary is generated in the current directory. It can be used immediately. Next up is a demonstration of best practices.

Best practice scenario 1: mirror Migration between different Cloud vendors

Take the migration from Aliyun Image Repository ACR to Tencent Cloud Image Warehouse TCR as an example.

1. Prepare the access credential information file for Tencent Cloud Image Warehouse TCR and Aliyun Image Warehouse ACR: auth.json grant-test.tencentcloudcr.com: username: xxx password: xxx grant-test2.tencentcloudcr.com: username: xxx password: xxx registry.cn-hangzhou.aliyuncs.com: username: xxx password: username: xxx password: xxx registry.hub.docker.com: username: xxx password: xxx

The configuration is simple. Enter the address of the source image repository and the address of the destination image repository. And enter the user name and password of the corresponding image repository.

Insecure indicates whether registry is a http service. If so, the insecure field needs to be true, and the default is false. Optional.

The user of the destination image repository needs to have the permission to push and create the repository. If it is not provided, the default is anonymous access.

The access credentials of Tencent Cloud TCR are obtained as follows: the access credentials of Ali Cloud Image Warehouse ACR are obtained as follows:

two。 Prepare the image rule file to be migrated: rule.yamlregistry.cn-hangzhou.aliyuncs.com/grantzhao/sichenzhao:xx ":" grant-test.tencentcloudcr.com/grantzhao/sichenzhao

This file is the source image and destination image that the configuration needs to transfer. The file rule is: source image address: destination image address, in which the source image address can be specified tag, tag may not be specified, or multiple tag can be specified. When you specify a single tag: the destination address may or may not contain a tag. If the tag is not included, the tag of the source mirror is used. When tag is not specified: the destination address must contain tag. When specifying multiple tag: multiple tag are separated by English commas, such as grant-test.tencentcloudcr.com/grantzhao/sichenzhao:1.0,2.0,3.0. At this point, the destination address cannot contain tag, and the tag of the source address is used by default.

3. Run the tool. / image-transfer-routines=5-securityFile=./security.yaml-ruleFile=./rule.yaml-ns=default\-registry=grant-test.tencentcloudcr.com-retry=2-qps=100

Parameter explanation:

-- ns specifies a default ns. If the ns of the destination warehouse is empty, it will be replaced by the default ns.

-- registry specifies a default registry. If the registry of the destination warehouse is empty, it will be replaced by the default registry.

-- routines=5, indicating that the number of concurrency is set to 5. The default is 5.

-- retry=2, which means that the number of retries after failure is 2, and the default is 2.

-- securityFile, specify the authentication file.

-- ruleFile, specify the image repository configuration file.

-- qps. Limit the qps of the request to no more than 100max s.

4. Running result

Last line

# Finished, 0 transfer jobs failed, 0 jobs generate failed #

Indicates that the operation was successful.

Best practice scenario 2: open source / self-built image warehouse migration to the cloud

Take the migration from open source image repository docker hub to Tencent Cloud image warehouse TCR as an example.

1. Prepare the access credential information file for docker hub and Tencent Cloud Image Warehouse TCR: security.yaml grant-test2.tencentcloudcr.com: username: xxx password: xxx registry.hub.docker.com: username: xxx password: xxx2. Prepare the image rule file to be migrated: image.json sichenzhao/private-test:xxx ":" grant-test2.tencentcloudcr.com/grantzhao/sichenzhao3. Run the tool. / image-transfer-routines=5-securityFile=./security.yaml-ruleFile=./rule.yaml-ns=default\-registry=grant-test.tencentcloudcr.com-retry=24. Running result

Last line

# Finished, 0 transfer jobs failed, 0 jobs generate failed #

Indicates that the operation was successful.

Best practice scenario 3: Tencent Cloud TCR personal Edition (CCR) migrates to Tencent Cloud Mirror Warehouse Enterprise Edition TCR with one click

The usage in this scenario is slightly different from that in the above two scenarios. The main performance is the change of input parameters.

1. Prepare the image authentication configuration file security.yaml grant-test.tencentcloudcr.com: username: xxx password: xxx grant-test2.tencentcloudcr.com: username: xxx password: xxx ccr.ccs.tencentyun.com: username: xxx password: xxx2. Prepare Tencent Cloud secret configuration file secret.yaml

For the one-click migration mode of TCR, the user name and password of the repository are not required as access authentication, but through the secret information of Tencent Cloud.

Ccr: secretId: xxx secretKey: xxx tcr: secretId: xxx secretKey: xxx

Note: the file format is shown above, and only secretId and secretKey entries are allowed to be modified.

If there is no secret information for ccr, tcr will be used instead. On the contrary, if there is no secret information for tcr, it will be replaced by ccr.

The secret information is obtained as follows: it contains secretid and secretkey information.

3. Run the tool

The parameter input here is slightly different from the above two scenarios.

. / image-transfer-- ccrToTcr=true-- routines=5-- securityFile=./security.yaml-- secretFile=./secret.yaml-- tcrName=tcr-test\-- retry=3-- tcrRegion=ap-guangzhou-- ccrRegion=ap-guangzhou-- qps=3000

Parameter explanation:

-- ccrToTcr=true, which means enabling the full migration mode of TCR with one button.

-- secretFile, which provides the secret.yaml configuration file.

-- tcrName=tcr-test, specifying the name of the destination tcr repository.

-- tcrRegion, specify the region where the destination tcr repository is located.

-- ccrRegion, specify the region where the source ccr repository is located.

4. Running result

One-click batch migration takes a long time because all the images of ccr need to be transferred to tcr.

Finally, you can see that 16 job failed. The tool finally lists the source and destination mirror addresses of the failed job. For these failed job, go to the warehouse to check and find that the tag of these job is invalid. So the transmission failed.

This is the end of the article on "how to use image-transfer". Thank you for reading! I believe you all have a certain understanding of the knowledge of "how to use image-transfer". If you want to learn more knowledge, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.