Shulou(Shulou.com)05/31 Report--

This article introduces how to use Velero to migrate cluster resources to TKE across cloud platforms. The content is very detailed. Interested friends can use it for reference. I hope it will be helpful to you.

Overview

Velero is a very powerful open source tool that can securely back up and restore, perform disaster recovery and migrate Kubernetes cluster resources and persistent volumes. You can use Velero to back up, restore and migrate cluster resources on the TKE platform. About how to use, see using object Storage COS as Velero Storage for backup and restore of Cluster Resources and using Velero Migration to replicate Cluster Resources in TKE This article describes how to use Velero to seamlessly migrate Kubernetes clusters from self-built or other cloud platforms to TKE platforms.

Migration principle

The architecture principle is similar to the principle of using Velero migration to replicate cluster resources. Both the migrated cluster and the migrated cluster install Velero instances and specify the same Tencent Cloud COS backend storage. The migrated cluster performs backups on demand, and the target cluster restores cluster resources as needed to achieve resource migration. The difference is that when you migrate cluster resources from self-built or other cloud platforms to TKE, you need to consider and solve the differences in cluster environment caused by cross-platform. Fortunately, Velero provides many practical backup and restore strategies to help us solve these problems. The following migration examples will show you how to make better use of them.

Environmental preparation

There are self-built or other cloud platform Kubernetes clusters (hereinafter referred to as cluster A), and the cluster version is above 1.10.

The TKE cluster for which the migration target has been created (hereinafter referred to as cluster B). For more information on creating a TKE cluster, see create a cluster.

Both cluster An and cluster B need to install Velero instances (version 1.5 or above) and share the same Tencent Cloud COS bucket as Velero backend storage. For installation steps, please see configure Storage and install Velero.

Ensure that the image resources can be pulled normally after migration.

Make sure that the K8S versions of the two clusters are API compatible, preferably the same version.

Migration guidance

Before the migration work, you should first clarify the migration ideas and make a detailed migration plan. The migration process probably needs to consider the following points:

Filter and analyze which cluster resources need to be migrated and which cluster resources do not need to be migrated

Filter and classify the list of resources that need to be migrated and the list of resources that do not need to be migrated according to the actual situation.

Consider whether you need to customize some Hook operations according to the business scenario

You need to consider whether the backup Hooks needs to be performed during the backup when backing up the cluster resources, such as the scenario where the in-memory data of the running application needs to be unloaded.

Similarly, when restoring (migrating) cluster resources, whether the restore Hooks needs to be performed during the restore, for example, some initialization work needs to be done before the restore.

Write backup and restore commands or resource lists as needed

Write backup and restore strategies according to the filtered and classified resource list. It is recommended to create a resource list to perform backup and restore in complex scenarios. YAML resource list is intuitive and easy to maintain, and the method specified by parameters can be used in simple migration scenarios or testing.

Dealing with differences in resources across cloud platforms

Since it is a cross-cloud platform, the dynamic creation of PVC storage classes and other relationships may be different. You need to plan in advance whether the dynamic PVC/PV storage class relationships need to be remapped, and you need to create a ConfigMap configuration of the relevant mapping before the restore operation. If you need to resolve more personalized differences, you can manually modify the resource list after the backup.

Verify the migration resources after the operation is completed

Check that the migrated cluster resources are as expected and that the data is fully available.

Operation steps

Next, we will demonstrate the steps of migrating resources from a cloud platform cluster A to TKE cluster B, which involves the practical basic knowledge of Velero backup and restore. If you do not understand, please go to [practical knowledge of Velero backup / restore] at the end of the article.

Create cluster A sample resources

The Nginx workload of PVC is included in the example of deploying Velero in cluster An of a cloud platform. To create PVC and PV directly using dynamic storage classes for convenience, first check the information of dynamic storage classes supported by the current cluster:

# get the storage class information supported by the current cluster, where xxx-StorageClass is the storage class name and xxx-Provider is the provider name, the same below. [root@iZj6c3vzs170hmeiu98h6aZ] # kubectl get scNAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGExxx-StorageClass xxx-Provider Delete Immediate true 3d3h...

Use the storage class named "xxx-StorageClass" in the cluster to create dynamically, and modify the PVC resource list of with-pv.yaml as shown below:

.-kind: PersistentVolumeClaimapiVersion: v1metadata: name: nginx-logs namespace: nginx-example labels: app: nginxspec: # Optional: modify the value of the storage class of PVC to a cloud platform storageClassName: xxx-StorageClass accessModes:-ReadWriteOnce resources: requests: storage: 20Gi # because the cloud platform limits storage to a minimum of 20Gi, this example needs to synchronously modify this value to 20GI.

After modification, the YAML in the application example creates the following cluster resources (nginx-example namespace):

[root@iZj6c3vzs170hmeiu98h6aZ nginx-app] # kubectl apply-f with-pv.yaml namespace/nginx-example createdpersistentvolumeclaim/nginx-logs createddeployment.apps/nginx-deployment createdservice/my-nginx created

The created PVC "nginx-logs" has been mounted to the / var/log/nginx directory of the nginx container as the log storage of the service. This example accesses the Nginx service in a browser test to produce some log data for the mounted PVC (for data comparison after subsequent restore).

# View the Nginx log size generated by the test, which is currently 84 K [root@iZj6c8ttj5dmmrs75yb7ybZ] # kubectl exec-it nginx-deployment-5ccc99bffb-6nm5w bash-n nginx-examplekubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD]-- [COMMAND] Defaulting container name to nginx.Use 'kubectl describe pod/nginx-deployment-5ccc99bffb-6nm5w-n nginx-example' to see all of the containers in this pod root@nginx-deployment-5ccc99bffb-6nm5w:/# du-sh / var/log/nginx/84K / var/log/nginx/# View the first two logs of accss.log and error.log root@nginx-deployment-5ccc99bffb-6nm5w:/# head-n 2 / var/log/nginx/access.log 192.168.0.73-[29/Dec/2020:03:02:31 + 0000] "GET /? spm=5176.2020520152.0.0.22d016ddHXZumX HTTP/1.1" 200612 "-" Mozilla/5.0 (Macintosh) Intel Mac OS X 10: 15: 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 "-" 192.168.0.73-- [29/Dec/2020:03:02:32 + 0000] "GET / favicon.ico HTTP/1.1" 404 555 "http://47.242.233.22/?spm=5176.2020520152.0.0.22d016ddHXZumX"" Mozilla/5.0 (Macintosh) Intel Mac OS X 10: 15: 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 "-" root@nginx-deployment-5ccc99bffb-6nm5w:/# head-n 2 / var/log/nginx/error.log 2020-12-29 03:02:32 [error] 6: * 597 open () "/ usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 192.168.0.73, server: localhost Request: "GET / favicon.ico HTTP/1.1", host: "47.242.233.22", referrer: "http://47.242.233.22/?spm=5176.2020520152.0.0.22d016ddHXZumX"2020/12/29 03:07:21 [error] 6: * 1172 open ()" / usr/share/nginx/html/0bef "failed (2: No such file or directory), client: 192.168.0.73, server: localhost Request: "GET / 0bef HTTP/1.0" confirms the list of resources to be migrated

Use the following command to output a list of all resources in the current cluster:

Kubectl api-resources-- verbs=list-o name | xargs-n 1 kubectl get-- show-kind-- ignore-not-found-- all-namespaces

You can also narrow the output of resources based on whether resources are namespaced or not:

View a namespace-insensitive resource inventory list:

Kubectl api-resources-namespaced=false-verbs=list-o name | xargs-n 1 kubectl get-- show-kind-- ignore-not-found

View the namespace-sensitive resource inventory list:

Kubectl api-resources-- namespaced=true-- verbs=list-o name | xargs-N1 kubectl get-- show-kind-- ignore-not-found-- all-namespaces

You can filter out the list of resources that need to be migrated according to the actual situation. In this example, resources related to Nginx workload under the "nginx-example" namespace are migrated directly from the cloud platform to the TKE platform, as shown below:

[root@iZj6c3vzs170hmeiu98h6aZ ~] # kubectl get all-n nginx-exampleNAME READY STATUS RESTARTS AGEpod/nginx-deployment-5ccc99bffb-tn2sh 2 Running 0 2d19hNAME TYPE CLUSTER-IP EXTERNAL-IP PORT (S) AGEservice/my-nginx LoadBalancer 172.21.1.185 x.x.x.x 80 Running 31455 / TCP 2d19hNAME READY UP-TO-DATE AVAILABLE AGEdeployment.apps/nginx-deployment 1 kubectl get pvc 1 1 1 2d19hNAME DESIRED CURRENT READY AGEreplicaset.apps/nginx-deployment-5ccc99bffb 1 1 1 2d19h [root@iZj6c3vzs170hmeiu98h6aZ ~] # kubectl get pvc-n nginx-exampleNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGEnginx-logs Bound d-j6ccrq4k1moziu1l6l5r 20Gi RWO xxx-StorageClass 2d19h [root@iZj6c3vzs170hmeiu98h6aZ ~] # kubectl get pvNAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGEd-j6ccrq4k1moziu1l6l5r 20Gi RWO Delete Bound nginx-example/nginx-logs xxx-StorageClass 2d19h confirms Hook policy

This example has been configured in with-pv.yaml to set the file system to read-only before backing up the Nginx workload, and to restore the read-write Hook policy after backup, as shown in the following YAML:

... Note to annotations: # backup Hook policy: set the nginx log directory to read-only mode before starting the backup Restore the read-write mode pre.hook.backup.velero.io/container: fsfreeze pre.hook.backup.velero.io/command:'["/ sbin/fsfreeze", "--freeze", "/ var/log/nginx"] 'post.hook.backup.velero.io/container: fsfreeze post.hook.backup.velero.io/command:' ["/ sbin/fsfreeze", "--unfreeze" after the backup is completed / var/log/nginx] 'spec: volumes:-name: nginx-logs persistentVolumeClaim: claimName: nginx-logs containers:-image: nginx:1.17.6 name: nginx ports:-containerPort: 80 volumeMounts:-mountPath: "/ var/log/nginx" name: nginx-logs ReadOnly: false-image: ubuntu:bionic name: fsfreeze securityContext: privileged: true volumeMounts:-mountPath: "/ var/log/nginx" name: nginx-logs. Start the migration operation

Next, write backup and restore strategies according to the actual situation, and begin to migrate the resources related to the Nginx workload of the cloud platform.

Perform a backup in cluster A

This example creates the following YAML to back up the resources you want to migrate:

ApiVersion: velero.io/v1kind: Backupmetadata: name: migrate-backup # must be a velero installed namespace namespace: velerospec: # resources containing only nginx-example namespaces includedNamespaces:-nginx-example # contains namespace-insensitive resources includeClusterResources: true # backup data storage location specified storageLocation: default # volume snapshot storage location specified volumeSnapshotLocations:-default # backup volumes defaultVolumesToRestic: true using restic

The backup process is as follows. When the backup status is "Completed" and the number of errors is 0, the backup process is complete:

[root@iZj6c8ttj5dmmrs75yb7ybZ] # kubectl apply-f backup.yaml backup.velero.io/migrate-backup created [root@iZj6c8ttj5dmmrs75yb7ybZ ~] # velero backup get NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTORmigrate-backup InProgress 00 2020-12-29 19:24:12 + 0800 CST 29d default [rootftiZi6c8tti5dmmrs75yb7vbZ ~ velero backup get NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTORmigrate-backup Completed 00 2020-12-29 19:24:28 + 0800 CST 29d default

"after the backup completes, temporarily update the backup storage location to read-only mode (optional, which prevents Velero from creating or deleting backup objects in the backup storage location during the restore process):"

Kubectl patch backupstoragelocation default-- namespace velero\-- type merge\-- patch'{"spec": {"accessMode": "ReadOnly"}} 'handles differences in resources across cloud platforms

Due to the differences in the dynamic storage classes used, you need the ConfigMap shown below to create a dynamic storage class name mapping for the persistent volume "nginx-logs":

ApiVersion: v1kind: ConfigMapmetadata: name: change-storage-class-config namespace: velero labels: velero.io/plugin-config: "" velero.io/change-storage-class: RestoreItemActiondata: # Storage class name is mapped to Tencent Cloud dynamic storage class cbs xxx-StorageClass: cbs

Apply the above ConfigMap configuration:

[root@VM-20-5-tlinux ~] # kubectl apply-f cm-storage-class.yaml configmap/change-storage-class-config created

The resource list of Velero backup is stored in object storage in json format. If you need more personalized migration, you can download the backup file directly and customize the modification. This example adds a "jokey-test:jokey-test" annotation to the Deployment resource customization of Nginx. The modification process is as follows:

Jokey@JOKEYLI-MB0 Downloads% mkdir migrate-backup# extract the backup file jokey@JOKEYLI-MB0 Downloads% tar-zxvf migrate-backup.tar.gz-C migrate-backup# to edit and modify the resources you want to customize. This example is nginx's Deployment resource to add the annotation item "jokey-test": "jokey-test" jokey@JOKEYLI-MB0 migrate-backup% cat resources/deployments.apps/namespaces/nginx-example/nginx-deployment.json {"apiVersion": "apps/v1", "kind": "Deployment" "metadata": {"annotations": {"jokey-test": "jokey-test",. # repackage the modified backup file jokey@JOKEYLI-MB0 migrate-backup% tar-zcvf migrate-backup.tar.gz *

Upload and replace the original backup file after custom modification and repackaging:

Perform a restore in cluster B

This example applies the resource list shown below to perform a restore operation (migration):

ApiVersion: velero.io/v1kind: Restoremetadata: name: migrate-restore namespace: velerospec: backupName: migrate-backup includedNamespaces:-nginx-example # enter the resource types that need to be restored as needed. There are no resources to be excluded under the nginx-example namespace, so write'* 'includedResources: -' * 'includeClusterResources: null # resources that are not included in the restore, and StorageClasses resource types are excluded. ExcludedResources:-storageclasses.storage.k8s.io # uses the labelSelector selector to select resources with a specific label, and since label selector filtering is no longer required in this example, it is annotated here. # labelSelector: # matchLabels: # app: nginx # set namespace relation mapping policy namespaceMapping: nginx-example: default restorePVs: true

The restore process is shown below. When the restore state is displayed as "Completed" and the number of "errors" is 0, the restore process is complete:

[root@VM-20-5-tlinux ~] # kubectl apply-f restore.yaml restore.velero.io/migrate-restore created [root @ VM-20-5-tlinux ~] # velero restore getNAME BACKUP STATUS STARTED COMPLETED ERRORS WARNINGS CREATED SELECTORmigrate-restore migrate-backup Completed 2021-01-12 20: 39:14 + 0800 CST 2021-01-12 20:39:17 + 0800 CST 00 2021-01-12 20:39:14 + 0800 CST Migration Resource Verification

First check to see if the state of the migrated resource is normal.

# because the "nginx-example" namespace was specified at restore time to map to the "default" namespace So the restored resource will run under the "default" namespace [root@VM-20-5-tlinux ~] # kubectl get all-n default NAME READY STATUS RESTARTS AGEpod/nginx-deployment-5ccc99bffb-6nm5w 2 Running 0 49sNAME TYPE CLUSTER-IP EXTERNAL-IP PORT (S) AGEservice/kube-user LoadBalancer 172.16.253.216 10.0.0.28 443:30060/TCP 8dservice/kubernetes ClusterIP 172.16.252.1 443/TCP 8dservice/my-nginx LoadBalancer 172.16.254.16 x.x.x.x 80:30840/TCP 49sNAME READY UP-TO-DATE AVAILABLE AGEdeployment.apps/nginx-deployment 1/1 1 1 49sNAME DESIRED CURRENT READY AGEreplicaset.apps/nginx-deployment-5ccc99bffb 1 1 1 49s

From the above, you can see that the running state of the migrated resources is normal, and then check whether the set restore policy is successful.

Verify that the dynamic storage class name mapping is correct:

# you can see that the storage class of PVC/PV is already "cbs", indicating that the storage class has been mapped successfully. [root@VM-20-5-tlinux ~] # kubectl get pvc- n default NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGEnginx-logs Bound pvc-bcc17ccd-ec3e-4d27-bec6-b0c8f1c2fa9c 20Gi RWO cbs 55s [root @ VM-20-5-tlinux ~] # kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGEpvc-bcc17ccd-ec3e-4d27-bec6-b0c8f1c2fa9c 20Gi RWO Delete Bound default/nginx-logs cbs 57s

Check to see if the "jokey-test" comment added to the "deployment.apps/nginx-deployment" customization before the restore was successful:

# the annotation "jokey-test" was obtained successfully, which means that the custom modification of resources is successful. [root@VM-20-5-tlinux ~] # kubectl get deployment.apps/nginx-deployment-o custom-columns=annotations:.metadata.annotations.jokey-testannotationsjokey-test

From the above view of the running status of the resource, you can see that the namespace mapping configuration is also successful.

Check whether the PVC data mounted by the workload is migrated successfully:

# check the data size in the mounted PVC data directory, which shows that 88K is more than that before migration. This is due to some logs generated by health checks initiated by Tencent Cloud CLB. [root@VM-20-5-tlinux ~] # kubectl exec-it nginx-deployment-5ccc99bffb-6nm5w-n default-- bashDefaulting container name to nginx.Use 'kubectl describe pod/nginx-deployment-5ccc99bffb-6nm5w-n default' to see all of the containers in this pod.root@nginx-deployment-5ccc99bffb-6nm5w:/# du-sh / var/log/nginx 88K / var/log/nginx# view the first two log messages, which are the same as before migration Generally speaking, PVC data is not lost root@nginx-deployment-5ccc99bffb-6nm5w:/# head-n 2 / var/log/nginx/access.log 192.168.0.73-- [29/Dec/2020:03:02:31 + 0000] "GET /? spm=5176.2020520152.0.0.22d016ddHXZumX HTTP/1.1" 200612 "-" Mozilla/5.0 (Macintosh) Intel Mac OS X 10: 15: 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 "-" 192.168.0.73-- [29/Dec/2020:03:02:32 + 0000] "GET / favicon.ico HTTP/1.1" 404 555 "http://47.242.233.22/?spm=5176.2020520152.0.0.22d016ddHXZumX"" Mozilla/5.0 (Macintosh) Intel Mac OS X 10: 15: 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 "-" root@nginx-deployment-5ccc99bffb-6nm5w:/# head-n 2 / var/log/nginx/error.log 2020-12-29 03:02:32 [error] 6: * 597 open () "/ usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 192.168.0.73, server: localhost Request: "GET / favicon.ico HTTP/1.1", host: "47.242.233.22", referrer: "http://47.242.233.22/?spm=5176.2020520152.0.0.22d016ddHXZumX"2020/12/29 03:07:21 [error] 6: * 1172 open ()" / usr/share/nginx/html/0bef "failed (2: No such file or directory), client: 192.168.0.73, server: localhost, request:" GET / 0bef HTTP/1.0 "

To sum up, this example successfully migrates Nginx (nginx-example namespace) workload-related resources and data of a cloud platform cluster A to TKE cluster B (default namespace).

Summary

This example explains and demonstrates the common ideas and steps for the migration of cluster resources to TKE. If you encounter scenarios that are not covered during the actual migration, you are welcome to consult and discuss the migration solution.

Practical knowledge of Velero backup / restore

Velero provides many very useful backup and restore strategies, which are briefly combed as follows:

When no filtering option is used, Velero includes all objects in the backup or restore operation, and you can specify parameters to filter resources on demand during backup and restore:

Filter parameters that contain relationships:

Filter parameters that do not contain relationships:

For more information, see Resource filtering.

-- exclude-namespaces: specifies the list of namespaces to exclude

-- exclude-resources: specifies the list of resource objects to exclude.

Velero.io/exclude-from-backup=true: this configuration item configures the label property for the resource object, and resource objects that have added this label configuration item will be excluded.

-- include-resources: specifies the list of resource objects to include.

-- include-namespaces: specifies the list of namespaces to include.

-- include-cluster-resources: specify whether to include resources for the cluster.

-- selector: specifies that it contains resources that match the tag selector.

Perform some Hook operations during the backup, such as the need to unload the memory data before the backup. For more information, see backup Hooks.

Perform some Hook operations during the restore, such as determining whether component dependencies are available before the restore. For more information, see restore Hooks.

Configure the PVC/PV volume-dependent mapping relationship configuration during restore:

Please refer to the restore reference for details.

Configure PV / PVC storage class mapping.

Configure the PVC binding node mapping.

Restic backup volume configuration

Starting with version 1.5, Velero uses Restic by default to back up all pod volumes without having to comment on each pod individually, so it is recommended that you use Velero version 1.5 or later.

Prior to version 1.5, when Velero used restic to back up volumes, Restic had two ways to discover Pod volumes that needed to be backed up:

After the backup is complete, you can view the backup volume information:

Kubectl-n velero get podvolumebackups-l velero.io/backup-name=-o yaml

After the restore is complete, you can view the restore volume information:

Kubectl-n velero get podvolumerestores-l velero.io/restore-name=-o yaml

The Pod volume backup selection used contains comments (default):

Kubectl-n annotate backup.velero.io/backup-volumes=

The Pod volume backup selection used does not contain comments:

Kubectl-n annotate backup.velero.io/backup-volumes-excludes=

This is the end of how to use Velero to migrate cluster resources to TKE across cloud platforms. I hope the above content can be of some help and learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.