Shulou(Shulou.com)06/01 Report--

This article focuses on "how to solve the notready problem in go". Interested friends may wish to have a look at it. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn how to solve the notready problem in go.

Environment: [root@k8s-01 ing] # kubectl versionServer Version: version.Info {Major: "1", Minor: "21", GitVersion: "v1.21.0", GitCommit: "cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState: "clean", BuildDate: "2021-04-08T16:25:06Z", GoVersion: "go1.16.1", Compiler: "gc" Platform: "linux/amd64"} [root@k8s-01 ing] # kubectl version | grep server [root @ k8s-01 ing] # kubectl get nodeNAME STATUS ROLES AGE VERSIONk8s-01 Ready control-plane,master 18d v1.21.0k8s-02 Ready worker 18d v1.21.0k8s-03 Ready worker 18d v1.21.0:

The k8s-02 node is in the notready state. Check the terminating time point of the pod, which is 7 hours ago. Check the messages log, and the error report also starts 7 hours ago.

Troubleshooting process 1. Check network connectivity

And the node is in a single-pass state: master and other nodes can ping k8s-02 machines, and k8s-02 cannot ping other machines.

It is true that there is a problem with the network, so check that the pod status of calico is ok and calico-kube-controller has failed over.

Describe is as follows:

To grab the icmp packet of k8s-02, master received it but did not reply:

Tcpdump-I eth0 icmp and host 10.170.36.462, kubelet troubleshooting

There is no clue to the network investigation here, start to view kubelet and Baidu according to the content of describe:

Baidu is a firewall, shut down swap and other operations, but I do not have … It doesn't work to restart the node or restart kubelet!

View the log messages:

May 11 21:27:58 k8s-02 kubelet: I0511 21 Kubelet nodes not sync 27 Kubelet nodes not sync 58.469919 651 kubelet.go:461] "Kubelet nodes not sync" May 11 21:27:59 k8s-02 kubelet: I0511 21 Kubelet nodes not sync 27 May 59.469278 651 kubelet.go:461] "Kubelet nodes not sync" May 11 21:28:00 k8s-02 kubelet: I0511 21 21 Kubelet nodes not sync 28 Kubelet nodes not sync 00.469261 651 k8s-02 kubelet: E0511 21 Kind 28 event.go:273] Unable to write event:'& v1.Event {TypeMeta:v1.TypeMeta {Kind: " APIVersion: ""}, ObjectMeta:v1.ObjectMeta {Name: "k8s-02.167e04da740613a2", GenerateName: ", Namespace:" default ", SelfLink:", UID: ", ResourceVersion:", Generation:0, CreationTimestamp:v1.Time {Time:time.Time {wall:0x0, ext:0, loc: (* time.Location) (nil)}, DeletionTimestamp: (* v1.Time) (nil), DeletionGracePeriodSeconds: (* int64) (nil), Labels: mapstring] string (nil), Annotations: mapstring] string (nil) OwnerReferences: [] v1.OwnerReference (nil), Finalizers: [] string (nil), ClusterName: ", ManagedFields: [] v1.ManagedFieldsEntry (nil)}, InvolvedObject:v1.ObjectReference {Kind:" Node ", Namespace:", Name: "k8s-02", UID: "k8s-02", APIVersion: ", ResourceVersion:", FieldPath: "}, Reason:" NodeHasSufficientPID ", Message:" Node k8s-02 status is now: NodeHasSufficientPID ", Source:v1.EventSource {Component:" kubelet ", Host:" k8s-02 "} FirstTimestamp:v1.Time {Time:time.Time {wall:0xc01ebe130bad1ba2, ext:12673521163, loc: (* time.Location) (0x74ad9e0)}, LastTimestamp:v1.Time {Time:time.Time {wall:0xc01ebe130bad1ba2, ext:12673521163, loc: (* time.Location) (0x74ad9e0)}}, Count:1, Type: "Normal", EventTime:v1.MicroTime {Time:time.Time {wall:0x0, ext:0, loc: (* time.Location) (nil)}}, Series: (* v1.EventSeries) (nil), Action: "" Related: (* v1.ObjectReference) (nil), ReportingController: "" ReportingInstance: ""}': 'Post "https://10.170.2.32:6443/api/v1/namespaces/default/events": dial tcp 10.170.2.32 k8s-02 kubelet 6443: I timeout' (may retry after sleeping) May 11 21:28:01 k8s-02 kubelet: I0511 21 dial tcp 28 k8s-02 kubelet 01.469790 651 kubelet.go:461]" Kubelet nodes not sync "May 11 21:28:02 k8s-02 kubelet: I0511 21 dial tcp 28Fr02.406629 651 kubelet. Go:461] "Kubelet nodes not sync" May 11 21:28:02 k8s-02 kubelet: I0511 21 k8s-02 kubelet 28 kubelet.go:461 02.40666669 651 kubelet.go:461] "Kubelet nodes not sync" May 11 21:28:02 k8s-02 kubelet: I0511 21 May 28 k8s-02 kubelet 02.469338 651 kubelet.go:461] "Kubelet nodes not sync" May 11 21:28:03 k8s-02 kubelet: I0511 21k8s-02 kubelet 28k8s-02 kubelet 03.407443 651 kubelet.go:461] "Kubelet nodes not sync" May 11 21:28:03 k8s-02 kubelet: I0511 21 May 28 kubelet.go:461 03.469928 651 kubelet.go:461] "Kubelet nodes not sync" May 11 21:28:03 k8s-02 kubelet: I0511 21 May 28 k8s-02 kubelet 03.617223 651 trace.go:205] Trace [766683077]: "Reflector ListAndWatch" name:k8s.io/client-go/informers/factory.go:134 (11-May-2021 21 27 Switzerland 33.615) (total time: 30001ms): May 11 21:28:03 K8s-02 kubelet: Trace [766683077]: [30.001402015s] [30.001402015s] ENDMay 11 21:28:03 k8s-02 kubelet: E0511 21purge 28ENDMay 03.617257 651 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch * v1.Service: failed to list * v1.Service: Get "https://10.170.2.32:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.170.2.32: 6443: iUnip o timeoutMay 11 21:28:04 k8s-02 kubelet: I0511 21 kubelet.go:461 28 Kubelet nodes not sync 04.407120 651 kubelet.go:461] "Kubelet nodes not sync" May 11 21:28:04 k8s-02 kubelet: I0511 21 Kubelet nodes not sync 28 Kubelet nodes not sync 04.469376 651 kubelet.go:461] "Kubelet nodes not sync" May 11 21:28:05 k8s-02 kubelet: I0511 2128 Kubelet nodes not sync 05.407095 651 kubelet.go:461] "Kubelet nodes not sync" May 11 21:28:05 K8s-02 kubelet: I0511 21 May 28 k8s-02 kubelet 05.469475 651 kubelet.go:461] "Kubelet nodes not sync" May 11 21:28:05 k8s-02 kubelet: I0511 21 Kubelet nodes not sync 28 May 05.769847 651 trace.go:205] Trace [347094812]: "Reflector ListAndWatch" name:k8s.io/client-go/informers/factory.go:134 (11-May-2021 21 27 30000ms) (total time: 30000ms): May 11 21:28:05 k8s-02 kubelet: Trace [347094812]: [30.000987614s] [30.000987614s] ENDMay 11 21:28:05 k8s-02 kubelet: E0511 21VO 28 reflector.go:138 05.769907 651 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch * v1.CSIDriver: failed to list * v1.CSIDriver: Get "https://10.170.2.32:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 10.170.2.32: 6443: iUnip o timeoutMay 11 21:28:06 k8s-02 kubelet: I0511 21 Kubelet nodes not sync 28 kubelet.go:461 06.407171 651 kubelet.go:461] "Kubelet nodes not sync" May 11 21:28:06 k8s-02 kubelet: I0511 21 Kubelet nodes not sync 28 Kubelet nodes not sync 06.469821 651 kubelet.go:461] "Kubelet nodes not sync" May 11 21:28:06 k8s-02 kubelet: I0511 21 21 28 kubelet.go:461 06.469863 651 kubelet.go:461] "Kubelet nodes not sync" May 11 21:28:06 K8s-02 kubelet: E0511 21 err= 28 Error getting node 06.469887 651 kubelet.go:2298] "err=" nodes have not yet been read at least once Cannot construct node object "May 11 21:28:06 k8s-02 kubelet: I0511 21 k8s-02 kubelet 28 kubelet.go:461 06.570550 651 kubelet.go:461]" Kubelet nodes not sync "May 11 21:28:06 k8s-02 kubelet: I0511 21 kubelet.go:461 28 Kubelet nodes not sync 06.570599 651 kubelet.go:461]" Kubelet nodes not sync "May 11 21:28:07 k8s-02 kubelet: I0511 21 kubelet.go:461 2807.407416 651 kubelet.go:461]" Kubelet nodes not sync "May 11 21:28:07 K8s-02 kubelet: I0511 21 Kubelet nodes not sync 28 k8s-02 kubelet 07.571241 651 kubelet.go:461] "Kubelet nodes not sync" May 11 21:28:08 k8s-02 kubelet: I0511 21 21 14 28 kubelet.go:461 08.407052 651 kubelet.go:461] "Kubelet nodes not sync"

The log is also Baidu, in short, I just can't connect to master.

3. Check according to the point in time

View pod stop time:

At the same time, according to the log in messages, the start error time is the same as the pod stop time, so check what actions have been done at that time, and then check for recovery!

4. Universal solution

Restart Dafa:

Restart node, restart kubelet is invalid!

Solution

Find the history command of the time point, do the externalIp operation, and be consistent with the ip of k8s-02, delete the externalIp network recovery, node ready!

= = replace ipvs with iptables and you can use externalIp normally! = =

At this point, I believe you have a deeper understanding of "how to solve the notready problem in go". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.