Shulou(Shulou.com)06/03 Report--

How to avoid frequent restart of container? I believe that most people have not yet learned this skill, in order to let you learn, to give you a summary of the following content, do not say much, let's move on.

Analysis:

When restart container is in Pod (specifically, when SyncPod () is executed periodically), Pod will find the last exit time of the current container through its own Status structure (because there may be multiple container in Pod), mark it as ts if it is the first restart, then restart container directly, and in Kubelet backOff.perItemBackoff (a map structure) Key records the time of the next backoff (the initial value is 10s, and then increases exponentially, the maximum 5min) based on the id calculated from the container and the pod object. If it is not the first restart, that is, the backOff record of this container already exists in the backOff.perItemBackoff of the Kubelet, which is counted as backoff, then if now ()-ts < backoff indicates that the waiting time is not enough, throw the CrashLoopBackOff Event (then wait until the next SyncPod cycle arrives. (re-compare this value) otherwise, it means it is time to wait for backoff and it is ready to restart. Execute backOff.Next () at this time, double the corresponding backoff of the container, and then perform the restart operation. In the process of calculating the backoff in step 3, it will also check the interval between the current time and the last container exit. If it is greater than 2 * MaxContainerBackOff = 10 minutes, then the backoff corresponding to this container will be reset to the initial value of 10s source code details.

Kubernetes/pkg/kubelet/kubelet.go

From the source code, we found that there are two constants in the kubernetes/pkg/kubelet/kubelet.go file:

MaxContainerBackOff = 300 * time.SecondbackOffPeriod = time.Second * 10

Use these two variables to construct a BackOff object, which is a property of kubelet and applies to all pod on the node

Klet.backOff = flowcontrol.NewBackOff (backOffPeriod, MaxContainerBackOff)

The BackOff structure is as follows

Type Backoff struct {sync.Mutex Clock clock.Clock defaultDuration time.Duration maxDuration time.Duration perItemBackoff map [string] * backoffEntry}

Then use this object in the SyncPod method

/ / Call the container runtime's SyncPod callbackresult: = kl.containerRuntime.SyncPod (pod, apiPodStatus, podStatus, pullSecrets, kl.backOff) kubernetes/pkg/kubelet/kuberuntime/kuberuntime_manager.go

The specific things SyncPod does are:

/ SyncPod syncs the running pod into the desired pod by executing following steps://// 1. Compute sandbox and container changes.// 2. Kill pod sandbox if necessary.// 3. Kill any containers that should not be running.// 4. Create sandbox if necessary.// 5. Create init containers.// 6. Create normal containers.func (m * kubeGenericRuntimeManager) SyncPod (pod * v1.Pod, _ v1.PodStatus, podStatus * kubecontainer.PodStatus, pullSecrets [] v1.Secret, backOff * flowcontrol.Backoff) (result kubecontainer.PodSyncResult) {

Also in this file, there is a key function

/ / If a container is still in backoff, the function will return a brief backoff error and// a detailed error message. Func (m * kubeGenericRuntimeManager) doBackOff (pod * v1.Pod, container * v1.Container, podStatus * kubecontainer.PodStatus, backOff * flowcontrol.Backoff) (bool, string, error) {var cStatus * kubecontainer.ContainerStatus for _ C: = range podStatus.ContainerStatuses {if c.Name = = container.Name & & c.State = = kubecontainer.ContainerStateExited {cStatus = c break}} if cStatus = = nil {return false, "", nil} glog.Infof ("checking backoff for container% q in pod% Q", container.Name Format.Pod (pod) / / Use the finished time of the latest exited container as the start point to calculate whether to do back-off. Ts: = cStatus.FinishedAt / / backOff requires a unique key to identify the container. Key: = getStableKey (pod, container) if backOff.IsInBackOffSince (key, ts) {if ref, err: = kubecontainer.GenerateContainerRef (pod, container) Err = = nil {m.recorder.Eventf (ref, v1.EventTypeWarning, events.BackOffStartContainer, "Back-off restarting failed container")} err: = fmt.Errorf ("Back-off% s restarting failed container=%s pod=%s", backOff.Get (key), container.Name, format.Pod (pod)) glog.Infof ("% s" Err.Error () return true, err.Error (), kubecontainer.ErrCrashLoopBackOff} backOff.Next (key, ts) return false, ", nil}

Where the backOff.Next function is defined as follows

/ / move backoff to the next mark, capping at maxDurationfunc (p * Backoff) Next (id string, eventTime time.Time) {p.Lock () defer p.Unlock () entry, ok: = p.perItemBackoff [id] if! ok | | hasExpired (eventTime, entry.lastUpdate P.maxDuration) {entry = p.initEntryUnsafe (id)} else {delay: = entry.backoff * 2 / / exponential entry.backoff = time.Duration (integer.Int64Min (int64 (delay), int64 (p.maxDuration)} entry.lastUpdate = p.Clock.Now ()}

After reading the above, have you mastered the ways to avoid frequent restart of container? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.