Shulou(Shulou.com)06/02 Report--

Kubernetes clusters usually use ssl certificates to encrypt communications, and Rancher automatically generates certificates for the cluster. In versions prior to Rancher v2.0.14 and v2.1.9, the automatically generated certificates for Rancher configuration clusters are valid for one year, which means that if you created a Rancher configuration cluster using these versions about a year ago, you need to start rotating certificates as soon as possible, otherwise the cluster will enter an error state after the certificate expires. The rotation certificate is an one-time operation, and the newly generated certificate is valid for 10 years.

This article will show you how to rotate certificates in detail. Even if your certificate has now expired, you can follow these steps to rotate the certificate. However, please note that do not upgrade rancher server, according to the last section of this article [the certificate has expired so that you cannot connect to k8s].

Note that when restarting a component, rotating Kubernetes certificates may cause your cluster to become temporarily unavailable. In addition, for production environments, it is recommended that you do this during the maintenance window.

Rotate certificates through UI (business cluster)

Note: available versions of Rancher v2.2.0 +

In Rancher v2.2.0 and later, cluster certificates can be updated through UI's certificate rotation feature, which is applicable to [custom installed clusters].

After the certificate rotation, the Kubernetes component will be restarted automatically. The restart will not affect the application Pod, and the restart time will take 3 to 5 minutes.

Certificate rotation can be used for the following services:

Etcd

Kubelet

Kube-apiserver

Kube-proxy

Kube-scheduler

Kube-controller-manager

The certificate is rotated through UI. Currently, it supports:

Update all service certificates in batch (CA certificate unchanged)

Update a specified service (CA certificate unchanged)

(important) Cluster update

If the Rancher version is upgraded from v2.x.x to 2.2.x, you need to do a cluster update first.

1. Enter [Global\ Cluster View]

2. Select "Ellipsis menu" to the right of "Target Cluster", and select upgrade.

3. Click "Show Advanced options" on the right to check whether the "Etcd Snapshot rotation" feature is enabled. It is recommended to enable this feature.

4. In "authorize Cluster access address", check whether the feature is enabled. It is recommended to start this feature. You do not need to enter the following domain name.

5. Finally, click "Save", and the cluster will be updated automatically.

Rotation certificate

1. Enter [Global\ Cluster View]

2. Select "Ellipsis menu" on the right side of the corresponding cluster, and select to update the certificate validity period.

3. Select to update all service certificates, and click Save

4. The cluster will automatically update the certificate

5. Because the certificate changes, the corresponding token will also change. After the cluster certificate is updated, the Pod connected to the API SERVER needs to be rebuilt to obtain a new token.

Cattle-system/cattle-cluster-agent

Cattle-system/cattle-node-agent

Cattle-system/kube-api-auth

Ingress-nginx/nginx-ingress-controller

Kube-system/canal

Kube-system/kube-dns

Kube-system/kube-dns-autoscaler

Other applications Pod

Rotate certificates through UI API (business cluster)

Note: available version Rancher v2.0.14 + v2.1.9+

For Rancher v2.0.14, v2.1.9, and later, the cluster certificate can be updated through API. API certificate rotation will update all component certificates at the same time, and the specified component is not supported to update the certificate.

1. In the "Global" view, locate the cluster that needs to update the certificate, then click the ellipsis menu on the right, and then click "API View".

2. Click the RotateCertificates on the upper right

3. Click Show Request

4. Click Send Request

5. Because the certificate changes, the corresponding token will also change. After the cluster certificate is updated, the Pod connected to the API SERVER needs to be rebuilt to obtain a new token.

Cattle-system/cattle-cluster-agent

Cattle-system/cattle-node-agent

Cattle-system/kube-api-auth

Ingress-nginx/nginx-ingress-controller

Kube-system/canal

Kube-system/kube-dns

Kube-system/kube-dns-autoscaler

Other applications Pod

RKE certificate rotation (common for local cluster and business cluster)

Note: available version rke v0.2.0+

If the Kubernetes cluster was previously created with rke v0.2.0, perform the rke up operation before rotating the certificate, please refer to:

Https://www.cnrancher.com/docs/rke/latest/cn/cert-mgmt/

The certificate is rotated through RKE. Currently, it supports:

Update all service certificates in batch (CA certificate unchanged)

Update a specified service (CA certificate unchanged)

Rotate CA and all service certificates

1. Update all service certificates in batches (CA certificates remain unchanged)

2. Update the specified service (the CA certificate remains unchanged)

3. Rotate CA and all service certificates

Rke cert rotate--rotate-caINFO [0000] Initiating Kubernetes clusterINFO [0000] Rotating Kubernetes cluster certificatesINFO [0000] [certificates] Generating CA kubernetes certificatesINFO [0000] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificatesINFO [0000] [certificates] Generating Kubernetes API server certificatesINFO [0000] [certificates] Generating Kube Controller certificatesINFO [0000] [certificates] Generating Kube Scheduler certificatesINFO [0000] [certificates] Generating Kube Proxy certificatesINFO [0000] [certificates] Generating Node certificateINFO [0001] [certificates] Generating admin certificates and kubeconfigINFO [0001] [certificates] Generating Kubernetes API server proxy client certificatesINFO [0001] [certificates] Generating etcd-xxxxx certificate and keyINFO [0001] [certificates] Generating etcd-yyyyy certificate and keyINFO [0001] [certificates] Generating etcd-zzzzz certificate and keyINFO [0001] Successfully Deployed state file at [. / cluster.rkestate] INFO [0001] Rebuilding Kubernetes cluster with rotated certificates

4. Because the certificate changes, the corresponding token will also change. After the cluster certificate update is completed, the Pod connected to the API SERVER needs to be rebuilt to obtain a new token.

Cattle-system/cattle-cluster-agent

Cattle-system/cattle-node-agent

Cattle-system/kube-api-auth

Ingress-nginx/nginx-ingress-controller

Kube-system/canal

Kube-system/kube-dns

Kube-system/kube-dns-autoscaler

Other applications Pod

Stand-alone container Rancher server certificate update

Rancher v2.0.14 +, v2.1.9 +, v2.2.0 + will automatically check the validity of the certificate, and if it is found that the certificate has expired, a new certificate will be automatically generated. So Rancher server running in a stand-alone container only needs to upgrade the rancher version to a supported version without doing anything else.

Fault handling

Prompt CA certificate is empty

If the following error occurs after updating the certificate, because the cluster update operation is not performed.

Solution method

1. Select the corresponding problem cluster, and then view the browser cluster ID, as shown below:

2. Execute the command kubectl edit clusters

If Rancher is installed by HA, execute the above command directly in the local cluster through the kube configuration file generated by rke

If Rancher is running in a single container, enter the container through docker exec-ti bash, then execute apt install vim-y to install the vim tool, and then execute the above command

3. Delete the configuration parameters under the spec.rancherKubernetesEngineConfig.rotateCertificates level:

Modify to

Input: the cluster will update automatically after wq saves the yaml file, and the certificate will be updated after the update is completed.

The certificate has expired so that K8S cannot be connected.

If the cluster certificate has expired, you cannot rotate the certificate even if you upgrade to Rancher v2.0.14, v2.1.9, and later. Rancher updates the certificate through Agent. If the certificate expires, you will not be able to connect with Agent.

Solution method

You can manually set the time of the node and adjust the time a little later. Because Agent only communicates with K8S master and Rancher server, if the rancher server certificate does not expire, you only need to adjust the K8S master node time.

Adjust the command:

Then upgrade the rancher server, then follow the certificate rotation steps to rotate the certificate, and then synchronize the time back after the certificate rotation is completed.

Check the validity period of the certificate

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.