Shulou(Shulou.com)05/31 Report--

This article shows you how to carry out etcd cluster operation and maintenance practice, the content is concise and easy to understand, it will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

Etcd is the data core of the Kubernetes cluster. In the most serious case, when there is a problem with etcd that cannot be recovered completely, the only way to solve the problem is to build a new environment. Therefore, the operation and maintenance knowledge related to etcd is more important. Etcd can be deployed as a container, or it can be built on the host. The following contents are general.

Backup and recovery of cluster

Add backup

#! / bin/bashIP=123.123.123.123BACKUP_DIR=/alauda/etcd_bak/mkdir-p $BACKUP_DIRexport ETCDCTL_API=3etcdctl-- endpoints= http://$IP:2379 snapshot save $BACKUP/snap-$ (date +% Y%m%d%H%M). Db# can recover the data of one node by backing up. In practice, in order to prevent abnormal nodes configured for scheduled tasks from not generating backups, it is recommended to add a few more.

Restore cluster

#! / bin/bash# uses etcdctl snapshot restore to generate data for each node # the key variable is #-- data-dir needs to be the data directory of the actual etcd runtime #-- name-- initial-advertise-peer-urls needs to use the configuration of each node #-- initial-cluster initial-cluster-token needs to be consistent with the original cluster ETCD_1=10.1.0.5ETCD_2=10.1.0.6ETCD_3=10.1.0 .7 for I in ETCD_1 ETCD_2 ETCD_3doexport ETCDCTL_API=3etcdctl snapshot restore snapshot.db-- data-dir=/var/lib/etcd-- name $I-- initial-cluster ${ETCD_1} = http://${ETCD_1}:2380, ${ETCD_2} = http://${ETCD_2}:2380,${ETCD_3}=http://${ETCD_3}:2380-- initial-cluster-token k8s_etcd_token-- initial-advertise-peer-urls http://$i:2380 & & mv / var/lib/etcd/ etcd_$idone# copy etcd_10.1.0.5 to node 10.1.0.5 Override / var/lib/etcd (same as-- data-dir path) # other nodes in turn and so on

SnapDb recovery created automatically with etcd

#! / bin/bash export ETCDCTL_API=3etcdctl snapshot restore snapshot.db-skip-hash-check-data-dir=/var/lib/etcd-name 10.1.0.5-initial-cluster 10.1.0.5 = http://10.1.0.5:2380,10.1.0.6=http://10.1.0.6:2380,10.1.0.7=http://10.1.0.7:2380-- initial-cluster-token k8s_etcd_token-- initial-advertise -peer-urls http://10.1.0.5:2380#: all nodes need to generate their own data directories. The only difference between the previous # and the previous command is that there are many-skip-hash-check (Skip Integrity check) # this method does not guarantee 100% recoverability, so it is recommended to add your own backup # you usually need to do some data compression and defragmentation after recovery, please refer to the corresponding section

Stepped on the pit

[version 3.0.14 etcd restore feature is not available] https://github.com/etcd-io/etcd/issues/7533 can use the updated etcd. Summary: recovery is to use DB to generate a copy of etcd data. With the same node, you can ensure that all data are the same except for the parameters specified during restore. This is why you use a DB for three (or five) times.

Cluster expansion-- from 1 to 3

Perform add

#! / bin/bashexport ETCDCTL_API=2etcdctl-endpoints= http://10.1.0.6:2379 member add 10.1.0.6 http://10.1.0.6:2380etcdctl-endpoints= http://10.1.0.7:2379 member add 10.1.0.7 http://10.1.0.7:2380# ETCD_NAME= "etcd_10.1.0.6" # ETCD_INITIAL_CLUSTER= "10.1.0.6 = http://10.1. 0.6 existing 2380 10.1.0.5 = http://10.1.0.5:2380"# ETCD_INITIAL_CLUSTER_STATE= "existing"

Node etcd parameter configuration to be added

#! / bin/bash/usr/local/bin/etcd-data-dir=/data.etcd-name 10.1.0.6--initial-advertise-peer-urls http://10.1.0.6:2380-listen-peer-urls http://10.1.0.6:2380-advertise-client-urls http://10.1.0.6:2379-listen-client-urls http://10.1.0.6:2379-initial-cluster 10.1.0.6 = http://10.1.0.6:2380,10.1.0.5=http://10.1.0.5:2380--initial-cluster-state exsiting--initial-cluster-token k8s_etcd_token#-- name=ip:peer_url# of all nodes in the initial-cluster cluster-- initial-cluster-state exsiting tells etcd that he belongs to an existing cluster Don't start your own door.

Stepped on the pit

From 1 to 3, the cluster will pass through a state of two nodes. At this time, the performance of the cluster may be like hanging up, and the commands such as endpoint status cannot be used, so we need to use member add to expand the cluster to three nodes, and then start the etcd instance in turn. This will ensure that etcd is healthy. From 3 to more, in fact, it is still member add, so don't worry about it.

Cluster plus certificate

Generate a certificate

Curl-s-L-o / usr/bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64curl-s-L-o / usr/bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64chmod + x / usr/bin/ {cfssl,cfssljson} cd / etc/kubernetes/pki/etcd

# cat ca-config.json {"signing": {"expiry": "100000h"}, "profiles": {"server": {"usages": ["signing", "key encipherment", "server auth", "client auth"], "expiry": "100000h"}, "client": {"usages": ["signing", "key encipherment", "server auth", "client auth"] "expiry": "100000h"}

# cat ca-csr.json {"CN": "etcd", "key": {"algo": "rsa", "size": 4096}, "names": [{"C": "CN", "L": "Beijing", "O": "Alauda", "OU": "PaaS", "ST": "Beijing"}]}

# cat server-csr.json {"CN": "etcd-server", "hosts": ["localhost", "0.0.0.0", "127.0.0.1", "all master nodes ip", "all master nodes ip", "all master nodes ip"], "key": {"algo": "rsa", "size": 4096}, "names": [{"C": "CN", "L": "Beijing" "O": "Alauda", "OU": "PaaS", "ST": "Beijing"}]}

# cat client-csr.json {"CN": "etcd-client", "hosts": ["]," key ": {" algo ":" rsa "," size ": 4096}," names ": [{" C ":" CN "," L ":" Beijing "," O ":" Alauda "," OU ":" PaaS "," ST ":" Beijing "}]}

Cd / etc/kubernetes/pki/etcdcfssl gencert-initca ca-csr.json | cfssljson-bare cacfssl gencert-ca=ca.pem-ca-key=ca-key.pem-config=ca-config.json-profile=server server-csr.json | cfssljson-bare servercfssl gencert-ca=ca.pem-ca-key=ca-key.pem-config=ca-config.json-profile=client client-csr.json | cfssljson-bare client

Reference link: https://lihaoquan.me/2017/3/29... .html

First update the peer-urls of the node

Export ETCDCTL_API=3etcdctl-- endpoints= http://x.x.x.x:2379 member list# 1111111111. # 2222222222. # 3333333333. Etcdctl-- endpoints= http://172.30.0.123:2379 member update 1111111111-- peer-urls= https://x.x.x.x:2380# executes three times to change the peer-urls of all three nodes to https

Modify configuration

# vim / etc/kubernetes/main*/etcd.yaml# etcd startup command part changes http to https Change the startup status to existing- advertise-client-urls= https://x.x.x.x:2379- initial-advertise-peer-urls= https://x.x.x.x:2380- initial-cluster=xxx= https://x.x.x.x:2380,xxx=https://x.x.x.x:2380, Xxx= https://x.x.x.x:2380- listen-client-urls= https://x.x.x.x:2379- listen-peer-urls= https://x.x.x.x:2380- initial-cluster-state=existing# etcd startup command part insert-cert-file=/etc/kubernetes/pki/etcd/server.pem- key-file=/etc/kubernetes/pki/etcd/server-key.pem- peer- Cert-file=/etc/kubernetes/pki/etcd/server.pem- peer-key-file=/etc/kubernetes/pki/etcd/server-key.pem- trusted-ca-file=/etc/kubernetes/pki/etcd/ca.pem- peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.pem- peer-client-cert-auth=true- client-cert-auth=true# search hostPath insert after-hostPath: path: / etc / kubernetes/pki/etcd type: DirectoryOrCreatename: etcd-certs# search mountPath is inserted after-mountPath: / etc/kubernetes/pki/etcd name: etcd-certs

# insert vim / etc/kubernetes/main*/kube-apiserver.yaml# apiserver startup part Change http to https--etcd-cafile=/etc/kubernetes/pki/etcd/ca.pem--etcd-certfile=/etc/kubernetes/pki/etcd/client.pem--etcd-keyfile=/etc/kubernetes/pki/etcd/client-key.pem--etcd-servers= https://x.x.x.x:2379,https://x.x.x.x:2379,https://x.x.x.x:2379

To sum up, prepare a set of certificates first. Then change the etcd intercom address to https, and the etcd log will report an error (which can be ignored), then start it with etcd-the parameter with certificate, and use the certificate in all the places where etcd is linked.

The pit encountered

[after etcd adds a certificate, the health check of apiserver is still a http request, and etcd will always scan the log] https://github.com/etcd-io/etcd/issues/9285

2018-02-06 12 error 41V 06.905234 I | embed: rejected connection from "127.0.0.1 EOF" (error "EOF", ServerName "")

Solution: remove the apiserver health check directly, or replace the default check command with curl (there should be no curl in the apiserver image. If it is a rigid requirement, re-build it yourself)

Cluster upgrade

Clusters that are already v3 do not need too much configuration, just keep the data directory and replace the image (or binary); the upgrade from v2 to v3 requires a merge operation, which I have not practiced and am not recommended.

Cluster status check

In fact, all of the above steps require the assistance of these commands--

#! / bin/bash# if you have a certificate, remove-- cert-- key-- cacert is fine #-- endpoints= needs to write the url of several nodes Endpoint status outputs a few messages export ETCDCTL_API=3etcdctl-- endpoints= https://x.x.x.x:2379-- cert=/etc/kubernetes/pki/etcd/client.pem-- key=/etc/kubernetes/pki/etcd/client-key.pem-- cacert=/etc/kubernetes/pki/etcd/ca.pem endpoint status-w tableetcdctl-- endpoints=xxxx endpoint healthetcdctl-- endpoints=xxxx member listkubectl get cs.

Data operations (delete, compression, defragmentation)

Delete

ETCDCTL_API=2 etcdctl rm-- recursive # v2 api can delete a "directory" ETCDCTL_API=3 etcdctl-- endpoints=xxx del / xxxxx-- prefix # v3 version # with a certificate, refer to the previous entry to add-- cert-- key-- cacert.

The pit encountered: in a customer environment, we found that there are a lot of "events" in the Kubernetes cluster, which is part of the events information seen by kubectl describe xxx. The data is so large that etcd is tired of running. We delete the useless data in this way.

Defragmentation

ETCDCTL_API=3 etcdctl-endpoints=xx:xx,xx:xx,xx:xx defragETCDCTL_API=3 etcdctl-endpoints=xx:xx,xx:xx,xx:xx endpoint status # look at the amount of data

Compress

ETCDCTL_API=3 etcdctl-endpoints=xx:xx,xx:xx,xx:xx compact# is not very useful in the etcd cluster that only uses K8s. I may not have encountered a specific scenario. # refer to this document # https://www.cnblogs.com/davygeek/p/8524477.html#, but it doesn't matter to run etcd-- auto-compaction-retention=1# adds this parameter to allow the etcd runtime to do its own compression.

common problem

Etcd is time-dependent, so the node time in the cluster must be synchronized with insufficient disk space. If the disk is eaten by etcd itself, you need to consider compressing and deleting data. After adding a certificate, all requests should be accompanied by a certificate, or you will prompt context deadline exceeded to be careful when indicating the status of the node in the etcd startup parameters when doing various operations, otherwise it will be troublesome to repeat the previous steps.

Log collection

Etcd logs only support syslog and stdout. Https://github.com/etcd-io/etcd/issues/7936etcd logs are very useful for troubleshooting. If we deploy etcd with host, the logs can be retrieved through systemd, but etcd started with kubeadm will lose all history after the container is restarted. We can do it with the following options--

Redirection of shell

Etcd-- xxxx-- xxxx > / var/log/etcd.log # cooperate with logratate to do log cutting # Mount the log to the host through volume

Supervisor

Supervisor has been an effective tool for keeping services running since containers first became popular.

Sidecar container (I'll add an example on GitHub later, github.com/jing2uo)

Sidecar simply means that there are multiple containers in a Pod (such as kubedns) that can see each other's processes, so we can use traditional strace to capture the output of etcd processes, and then operate like shell redirects in Sidecar containers.

Strace-e trace=write-s 200-f-p 1

Clusters deployed by Kubeadm 1.13

Recently, when we tested the Kubernetes 1.13 cluster, we found some interesting changes. The command above won't work at a glance-- https://kubernetes.io/docs/set. Logy/ distinguishes Stacked etcd topology from External etcd topology, and the official link to this picture is very vivid--

For etcd clusters in this mode, the most obvious difference is that the initial-cluster startup parameters of etcd in the container only have their own IP, so I am a little confused about how to restore it. In fact, the basic principle has not changed, Kubeadm hides a ConfigMap, and the startup parameters are put here--

Kubectl get cm etcdcfg-n kube-system-o yaml

Etcd: local: serverCertSANs:-"192.168.8.21" peerCertSANs:-"192.168.8.21" extraArgs: initial-cluster: 192.168.8.21 = https://192.168.8.21:2380,192.168.8.22=https://192.168.8.22:2380,192.168.8.20=https://192.168.8.20:2380 initial-cluster-state: new name: 192.168.8.21 listen-peer-urls: https://192 .168.8.21: 2380 listen-client-urls: https://192.168.8.21:2379 advertise-client-urls: https://192.168.8.21:2379 initial-advertise-peer-urls: https://192.168.8.21:2380

Quan A

Q: how do you do etcd monitoring and alarm? What are the warning items?

A: the alarm depends on what kind of monitoring is used. Prometheus and Grafana are more common in supporting Kubernetes. I have not specifically configured the alarm items. The points you can pay attention to are the amount of data you can see in endpoint status-w table, the health status of endpoints health, and the use of memory. For more information, please refer to Prometheus's exporter.

Q: is using Kubeadm to deploy highly available clusters equivalent to deploying three independent single points of Master first, and finally using etcd to add nodes to get through the data?

A: no, Kubeadm deployment will build an etcd cluster at the very beginning. You need to prepare etcd before apiserver starts, otherwise apiserver cannot start and clusters cannot communicate with each other. You can try to build a cluster manually, do not use Kubeadm, open the components one by one, and then you will have a better understanding of the component relationships of Kubernetes.

How to ensure the high availability of Q:etcd across computer rooms? Is there a good UI tool recommendation for managing etcd?

A:etcd has high requirements for time and network, so if the network across the computer room is not good, then the performance is very poor, just choose to enter a link description over there. I forgot to mention a mirror of etcd in sharing, you can refer to it. Across computer rooms, I think high-speed Internet is a prerequisite, but it hasn't been done yet. I haven't looked for the UI tool, it's all a command line operation.

Have you ever tried to restore the backup of etcd by kubectl on the etcd node in the cluster started by Q:Kubeadm?

A: kubectl has not been used to handle backup and recovery of etcd. The recovery of etcd depends on using SnapDb to generate the data directory and throw the etcd process into the container. Similar operations cannot be avoided, and the startup status needs to be modified. Etcd started by kubeadm can be queried and exec through kubectl, but data manipulation should not be possible, for example, when restoring etcd ing, how can it work if etcd,kubectl cannot be connected?

Q:kubeadm-ha starts 3 Master and has 3 etcd nodes. How to cluster with 3 etcd outside the cluster to make 3 Master and 6 etcd?

A: you can refer to the expansion section in the document. As long as you ensure that the parameters of etcd are correct, even if a cluster is partially containerized and part of the host is available (of course, this is not recommended). You can first build a cluster with kubeadm, and then add the other three nodes by expanding capacity, or build an etcd cluster before kubeadm operation. Then kubeadm calls it.

Q: have you tried the rolling upgrade of Kubeadm, the change of etcd version, the reboot of each Master machine, any abnormalities in data synchronization, etc.?

A: yes. Kubeadm's rolling upgrade company has a step-by-step upgrade from 1.7 to 1.11,1.12, which is more or less a bit of a pit, but today's topic is etcd, so I didn't mention this part. We have no problem with the consistency of the data after each Master is rebooted separately, and what is more extreme is to directly shut down the three Master for one day, and then restore it after startup.

The above content is how to carry out the practice of etcd cluster operation and maintenance. Have you learned any knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.