Shulou(Shulou.com)05/31 Report--

This article introduces the knowledge of "how to achieve SuperEdge distributed health check". Many people will encounter this dilemma in the operation of actual cases, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Preface

SuperEdge is an edge container management system based on native Kubernetes. The system extends the cloud native capabilities to the edge side, well realizes the management and control of the cloud to the edge end, and greatly simplifies the process of application deployment from the cloud to the edge side. At the same time, SuperEdge designs a distributed health check mechanism to avoid a large number of pod migration and reconstruction caused by the instability of the cloud network, and ensures the stability of the service.

In the edge computing scenario, the network environment between the edge node and the cloud is very complex, and the connection between the edge node and the cloud is unreliable. In the native Kubernetes cluster, the connection between the apiserver and the node is interrupted, and the node status is abnormal, which eventually leads to the expulsion of pod and the lack of endpoint, resulting in service interruption and fluctuation. Specifically, the native Kubernetes processing is as follows:

The missing node is set to the ConditionUnknown state and the taints of NoSchedule and NoExecute is added

The pod on the missing node is expelled and rebuilt on other nodes

The pod on the missing node is removed from the Endpoint list in Service

Therefore, the edge computing scenario only depends on the connection between the edge and the apiserver is not enough to judge whether the node is abnormal, it will cause misjudgment because of the unreliability of the network and affect the normal service. Compared with the connection between the cloud and the edge, it is obvious that the connection between the edge nodes is more stable and has higher reference value, so superedge proposed the edge distributed health check mechanism. In this mechanism, the node state judgment should not only consider the apiserver factors, but also introduce the node evaluation factors, and then make a more comprehensive state judgment of the node. Through this function, we can avoid a large number of pod migration and reconstruction caused by the unreliable cloud edge network, and ensure the stability of the service.

Specifically, the accuracy of node state judgment is enhanced mainly through the following three levels:

Each node regularly detects the health status of other nodes.

All nodes in the cluster vote regularly to determine the status of each node.

Cloud and edge nodes work together to determine node state

The final judgment of distributed health check is as follows:

Edge-health-daemon source code analysis

Before diving into the source code, let's introduce the implementation principle of distributed health check. The architecture diagram is as follows:

Kubernetes each node will correspond to a Lease object,kubelet under kube-node-lease namespace and update the Lease object of the corresponding node every node-status-update-frequency interval (default is 10s).

Node-controller checks whether the Lease object is updated every node-monitor-period time (default is 5s). If no update has occurred after the node-monitor-grace-period time (default is 40s), the node is considered unhealthy and the NodeStatus (ConditionUnknown) will be updated.

When the node heartbeat times out (ConditionUnknown), node controller adds the following taints to the node:

Spec:... Taints:-effect: NoSchedule key: node.kubernetes.io/unreachable timeAdded: "2020-07-02T03:50:47Z"-effect: NoExecute key: node.kubernetes.io/unreachable timeAdded: "2020-07-02T03:50:53Z"

At the same time, endpoint controller kicks off all pod on the mother machine from the endpoint backend

For the mother machine with NoSchedule taint, Scheduler will not schedule a new load on the node, while for the mother machine with NoExecute (node.kubernetes.io/unreachable) taint, node controller will expel the pod on the node for a period of time after the node heartbeat timeout (default 5mins)

The edge-health-daemon component at the side of the distributed health check performs a distributed health check on the edge nodes in the same area and sends the health status vote result to apiserver (annotation for node)

In addition, in order to achieve when the cloud edge is disconnected and the distributed health check status is normal:

The pod on the missing node will not be removed from Service's Endpoint list

The pod on the missing node will not be expelled

You also need to run edge-health-admission (Kubernetes mutating admission webhook) on the cloud, constantly adjust the node taint of the kube-controller-manager settings according to the node edge-health annotation (remove NoExecute taint) and endpoints (move the pods on the missing node from endpoint subsets notReadyAddresses to addresses), so that the cloud and the edge jointly determine the node state.

This chapter will mainly introduce the principle of edge-health-daemon, as follows: the relevant data structure of edge-health-daemon:

Type EdgeHealthMetadata struct {* NodeMetadata * CheckMetadata} type NodeMetadata struct {NodeList [] v1.Node sync.RWMutex} type CheckMetadata struct {CheckInfo map [string] CheckDetail / / Checker ip: {Checked ip:Check detail} CheckPluginScoreInfo map [string] float64 / / Checked ip: {Plugin name:Check score} sync.RWMutex} type CheckDetail struct {Normal bool Time time.Time} type CommunInfo struct {SourceIP string / / ClientIP Checker ip CheckDetail map[string] CheckDetail / / Checked ip:Check detail Hmac string}

The meaning is as follows:

NodeMetadata: edge node cache maintained to implement a sub-regional distributed health check mechanism, which contains a list of all edge nodes in the region NodeList

CheckMetadata: stores the results of a health check, specifically including two data structures:

CheckPluginScoreInfo: organized for Checked ip: {Plugin name:Check score}. The first-level key indicates that the checked ip; second-level key indicates that the name of the check plug-in; value indicates that the check score

CheckInfo: organized for Checker ip: {Checked ip:Check detail}. The first level key indicates that the second level key of the ip; performing the check indicates that the checked ip;value represents the check result CheckDetail.

CheckDetail: represents the result of a health check

If Normal:Normal is true, the check result is normal; false means abnormal.

Time: indicates the time when the result was obtained, used to judge the validity of the result (results that have not been updated for more than a period of time will be invalid)

CommunInfo: data used by edge nodes to send health check results to other nodes, including:

SourceIP: indicates the ip that performs the check

CheckDetail: organized for Checked ip:Check detail, including the checked ip and the check results

Hmac:SourceIP and CheckDetail are obtained by hmac, which are used to determine the validity of the transmitted data (whether it has been tampered with or not) in the process of edge node communication.

Edge-health-daemon subject logic includes four parts of functions:

SyncNodeList: refresh the node cache according to the zone where the edge node is located, and update the CheckMetadata related data at the same time

ExecuteCheck: execute several kinds of health check plug-ins (ping,kubelet, etc.) for each edge node, and summarize the check scores of each plug-in to get the result of whether the node is healthy according to the baseline set by the user.

Commun: send the health check results of this node to other nodes

Vote: classifies the health check results of all nodes. If a node is judged as normal by most of the nodes (> 1), superedgehealth/node-health:true annotation is added to the node, indicating that the distributed health check result of the node is normal; otherwise, adding superedgehealth/node-health:false annotation to the node indicates that the distributed health check result of the node is abnormal.

The following is a source code analysis of the above functions in turn:

1 、 SyncNodeList

SyncNodeList is executed every HealthCheckPeriod seconds (health-check-period option), and refreshes the node cache according to the following categories:

If there is no configmap named edge-health-zone-config under kube-system namespace, multi-region detection is not enabled, so the list of all edge nodes is obtained and the node cache is refreshed.

Otherwise, if the TaintZoneAdmission of the configmap data part of the edge-health-zone-config is false, multi-region detection is not enabled, so the list of all edge nodes is obtained and the node cache is refreshed.

If TaintZoneAdmission is true and node has a "superedgehealth/topology-zone" label (marked area), get the same list of nodes as "superedgehealth/topology-zone" label value and refresh node cache

If node does not have a "superedgehealth/topology-zone" label, only the edge nodes themselves are added to the list of distributed health check nodes and the node cache (only itself) is refreshed.

Func (ehd * EdgeHealthDaemon) SyncNodeList () {/ / Only sync nodes when self-located found var host * v1.Node if host = ehd.metadata.GetNodeByName (ehd.cfg.Node.HostName) Host = = nil {klog.Errorf ("Self-hostname% s not found", ehd.cfg.Node.HostName) return} / / Filter cloud nodes and retain edge ones masterRequirement, err: = labels.NewRequirement (common.MasterLabel, selection.DoesNotExist, [] string {}) if err! = nil {klog.Errorf ("New masterRequirement failed% + v") Err) return} masterSelector: = labels.NewSelector () masterSelector = masterSelector.Add (* masterRequirement) if mrc, err: = ehd.cmLister.ConfigMaps (metav1.NamespaceSystem) .Get (common.TaintZoneConfigMap) Err! = nil {if apierrors.IsNotFound (err) {/ / multi-region configmap not found if NodeList, err: = ehd.nodeLister.List (masterSelector) Err! = nil {klog.Errorf ("Multi-region configmap not found and get nodes err% + v", err) return} else {ehd.metadata.SetByNodeList (NodeList)}} else {klog.Errorf ("Get multi-region configmap err% + v") Err) return}} else {/ / multi-region configmap found mrcv: = mrc.Data [common.TaintZoneConfigMapKey] klog.V (4). Infof ("Multi-region value is% s", mrcv) if mrcv = = "false" {/ / close multi-region check if NodeList, err: = ehd.nodeLister.List (masterSelector) Err! = nil {klog.Errorf ("Multi-region configmap exist but disabled and get nodes err% + v", err) return} else {ehd.metadata.SetByNodeList (NodeList)}} else {/ / open multi-region check if hostZone, existed: = host.Labels [common.TopologyZone] Existed {klog.V (4) .Infof ("Host% s has HostZone% s", host.Name, hostZone) zoneRequirement, err: = labels.NewRequirement (common.TopologyZone, selection.Equals, [] string {hostZone}) if err! = nil {klog.Errorf ("New masterZoneRequirement failed:% + v" Err) return} masterZoneSelector: = labels.NewSelector () masterZoneSelector = masterZoneSelector.Add (* masterRequirement, * zoneRequirement) if nodeList, err: = ehd.nodeLister.List (masterZoneSelector) Err! = nil {klog.Errorf ("TopologyZone label for hostname% s but get nodes err:% + v", host.Name Err) return} else {ehd.metadata.SetByNodeList (nodeList)}} else {/ / Only check itself if there is no TopologyZone label klog.V (4) .Infof ("Only check itself since there is no TopologyZone label for hostname% s" Host.Name) ehd.metadata.SetByNodeList ([] * v1.Node {host})} / / Init check plugin score ipList: = make (map [string] struct {}) for _, node: = range ehd.metadata.Copy () {for _ Addr: = range node.Status.Addresses {if addr.Type = = v1.NodeInternalIP {ipList [addr.Address] = struct {} {} ehd.metadata.InitCheckPluginScore (addr.Address)} / / Delete redundant check plugin score for _, checkedIp: = range ehd.metadata.CopyCheckedIp () {if _, existed: = ipList [checkedIp] ! existed {ehd.metadata.DeleteCheckPluginScore (checkedIp)}} / / Delete redundant check info for checkerIp: = range ehd.metadata.CopyAll () {if _, existed: = ipList [checkerIp] ! existed {ehd.metadata.DeleteByIp (ehd.cfg.Node.LocalIp, checkerIp)}} klog.V (4). Infof ("SyncNodeList check info% + v successfully", ehd.metadata)}... func (cm * CheckMetadata) DeleteByIp (localIp, ip string) {cm.Lock () defer cm.Unlock () delete (cm.CheckInfo [localIp], ip) delete (cm.CheckInfo, ip)}

After updating the node cache according to the logic above, the CheckMetadata.CheckPluginScoreInfo is initialized and the node ip is assigned to the CheckPluginScoreInfo key (Checked ip: checked ip)

In addition, the redundant items in CheckMetadata.CheckPluginScoreInfo and CheckMetadata.CheckInfo will be deleted (outside the scope of the edge node inspection)

2 、 ExecuteCheck

ExecuteCheck is also executed every HealthCheckPeriod seconds (health-check-period option). Several kinds of health check plug-ins (ping,kubelet, etc.) are executed for each edge node, and the check scores of each plug-in are summarized to determine whether the node is healthy according to the baseline HealthCheckScoreLine (health-check-scoreline option) set by the user.

Func (ehd * EdgeHealthDaemon) ExecuteCheck () {util.ParallelizeUntil (context.TODO (), 16, len (ehd.checkPlugin.Plugins), func (index int) {ehd.checkPlugin.Plugins.CheckExecute (ehd.metadata.CheckMetadata)}) klog.V (4) .Infof ("CheckPluginScoreInfo is% + v after health check", ehd.metadata.CheckPluginScoreInfo) for checkedIp, pluginScores: = range ehd.metadata.CopyCheckPluginScore () {totalScore: = 0.0 for _ Score: = range pluginScores {totalScore + = score} if totalScore > = ehd.cfg.Check.HealthCheckScoreLine {ehd.metadata.SetByCheckDetail (ehd.cfg.Node.LocalIp, checkedIp, metadata.CheckDetail {Normal: true})} else {ehd.metadata.SetByCheckDetail (ehd.cfg.Node.LocalIp, checkedIp) Metadata.CheckDetail {Normal: false}} klog.V (4) .Infof ("CheckInfo is% + v after health check", ehd.metadata.CheckInfo)}

Here, ParallelizeUntil will be called to execute the check plug-ins concurrently. Edge-health currently supports ping and kubelet check plug-ins. In the checkplugin directory (github.com/superedge/superedge/pkg/edge-health/checkplugin), register to the PluginInfo singleton (plugin list) through Register, as follows:

/ TODO: handle flag parse errorsfunc (pcp * PingCheckPlugin) Set (s string) error {var err error for _, para: = range strings.Split (s, ",") {if len (para) = = 0 {continue} arr: = strings.Split (para, "=") trimKey: = strings.TrimSpace (arr [0]) switch trimKey {case "timeout": timeout _: = strconv.Atoi (strings.TrimSpace (arr [1]) pcp.HealthCheckoutTimeOut = timeout case "retries": retries, _: = strconv.Atoi (strings.TrimSpace (arr [1])) pcp.HealthCheckRetries = retries case "weight": weight, _: = strconv.ParseFloat (strings.TrimSpace (arr [1])) 64) pcp.Weight = weight case "port": port, _: = strconv.Atoi (strings.TrimSpace (arr [1])) pcp.Port = port}} PluginInfo = NewPlugin () PluginInfo.Register (pcp) return err} func (p * Plugin) Register (plugin CheckPlugin) {p.Plugins = append (p.Plugins Plugin) klog.V (4). Info ("Register check plugin:% + v", plugin)}... var (PluginOnce sync.Once PluginInfo Plugin) type Plugin struct {Plugins [] CheckPlugin} func NewPlugin () Plugin {PluginOnce.Do (func () {PluginInfo = Plugin {Plugins: [] CheckPlugin {},}}) return PluginInfo}

The logic for each plug-in to perform health check is encapsulated in CheckExecute. Here, take ping plugin as an example:

/ / github.com/superedge/superedge/pkg/edge-health/checkplugin/pingcheck.gofunc (pcp * PingCheckPlugin) CheckExecute (checkMetadata * metadata.CheckMetadata) {copyCheckedIp: = checkMetadata.CopyCheckedIp () util.ParallelizeUntil (context.TODO (), 16, len (copyCheckedIp), func (index int) {checkedIp: = copyCheckedIp [index] var err error for i: = 0; I

< pcp.HealthCheckRetries; i++ { if _, err := net.DialTimeout("tcp", checkedIp+":"+strconv.Itoa(pcp.Port), time.Duration(pcp.HealthCheckoutTimeOut)*time.Second); err == nil { break } } if err == nil { klog.V(4).Infof("Edge ping health check plugin %s for ip %s succeed", pcp.Name(), checkedIp) checkMetadata.SetByPluginScore(checkedIp, pcp.Name(), pcp.GetWeight(), common.CheckScoreMax) } else { klog.Warning("Edge ping health check plugin %s for ip %s failed, possible reason %s", pcp.Name(), checkedIp, err.Error()) checkMetadata.SetByPluginScore(checkedIp, pcp.Name(), pcp.GetWeight(), common.CheckScoreMin) } })}// CheckPluginScoreInfo relevant functionsfunc (cm *CheckMetadata) SetByPluginScore(checkedIp, pluginName string, weight float64, score int) { cm.Lock() defer cm.Unlock() if _, existed := cm.CheckPluginScoreInfo[checkedIp]; !existed { cm.CheckPluginScoreInfo[checkedIp] = make(map[string]float64) } cm.CheckPluginScoreInfo[checkedIp][pluginName] = float64(score) * weight} CheckExecute 会对同区域每个节点执行 ping 探测(net.DialTimeout)，如果失败，则给该节点打 CheckScoreMin 分(0)；否则，打 CheckScoreMax 分(100) 每种检查插件会有一个 Weight 参数，表示了该检查插件分数的权重值，所有权重参数之和应该为1，对应基准分数线 HealthCheckScoreLine 范围0-100。因此这里在设置分数时，会乘以权重 回到 ExecuteCheck 函数，在调用各插件执行健康检查得出权重分数(CheckPluginScoreInfo)后，还需要将该分数与基准线 HealthCheckScoreLine 对比：如果高于(>

=) fractional line, the node is considered to be normal this time; otherwise, it is abnormal.

Func (ehd * EdgeHealthDaemon) ExecuteCheck () {util.ParallelizeUntil (context.TODO (), 16, len (ehd.checkPlugin.Plugins), func (index int) {ehd.checkPlugin.Plugins.CheckExecute (ehd.metadata.CheckMetadata)}) klog.V (4) .Infof ("CheckPluginScoreInfo is% + v after health check", ehd.metadata.CheckPluginScoreInfo) for checkedIp, pluginScores: = range ehd.metadata.CopyCheckPluginScore () {totalScore: = 0.0 for _ Score: = range pluginScores {totalScore + = score} if totalScore > = ehd.cfg.Check.HealthCheckScoreLine {ehd.metadata.SetByCheckDetail (ehd.cfg.Node.LocalIp, checkedIp, metadata.CheckDetail {Normal: true})} else {ehd.metadata.SetByCheckDetail (ehd.cfg.Node.LocalIp, checkedIp) Metadata.CheckDetail {Normal: false}} klog.V (4) .Infof ("CheckInfo is% + v after health check", ehd.metadata.CheckInfo)}

3 、 Commun

After performing a health check on the edge nodes in the same region, you need to pass the results of the check to other nodes, which is what the commun module is responsible for:

Func (ehd * EdgeHealthDaemon) Run (stopCh

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.