Shulou(Shulou.com)06/02 Report--

This article shows you how to do abnormal k8s cpu limit node nodes. The content is concise and easy to understand. It will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

Log information:

OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused\" process_linux.go:367: setting cgroup config for procHooks process caused\\ "failed to write 100000 to cpu.cfs_period_us: write / sys/fs/cgroup/cpu,cpuacct/kubepods/burstable/pod0f117614-9bae-11e9-b422-525400530fa1/fa79711ac936f4650b00bab66e21f5efb64fac642aa38e9c314385ab4023b484/cpu.cfs_period_us: invalid argument\": unknown

From the description of log information: cgroup cput parameter: cpu.cfs_period_us: invalid argument\\ "\": there is a problem with unknown!

Parameter file directory: / sys/fs/cgroup/cpu,cpuacct/kubepods/burstable/pod0f117614-9bae-11e9-b422-525400530fa1/fa79711ac936f4650b00bab66e21f5efb64fac642aa38e9c314385ab4023b484/cpu.cfs_period_us

The specific reason is unknown!

Symptoms that occur:

The newly dispatched pod has started and reported the above error. There seems to be no problem with the previous pod.

Https://my.oschina.net/xiaominmin/blog/3068378

Https://kubernetes.io/zh/docs/tasks/administer-cluster/cpu-management-policies/

Https://qingwave.github.io/2019/01/09/%E6%B7%B1%E5%85%A5%E7%90%86%E8%A7%A3K8s%E8%B5%84%E6%BA%90%E9%99%90%E5%88%B6/#%E5%86%85%E5%AD%98%E9%99%90%E5%88%B6

Https://www.yangcs.net/posts/understanding-resource-limits-in-kubernetes-cpu-time/

Cause: kernel issu

Solution: node executes script

# / usr/bin/python2 "" Fixs a CFS bug in Container-optimized OS (COS) nodes.The bug is described at: https://docs.google.com/document/d/13KLD__6A935igLXpTFFomqfclATC89nAkhPOxsuKA0I/edit#Examples:To check script sanity in COS VM: sudo python fix-cfs.py-dry_runTo fix cfs bug in COS VM: sudo python fix-cfs.pyWhen running in a container/DaemonSet: sudo python fix-cfs.py-sys=/host/sys-dry_run sudo python fix-cfs.py -- sys=/host/sys sudo python fix-cfs.py-- sys=/host/sys-- interval=10 "" from _ _ future__ import print_functionimport argparseimport mathimport osimport timedef _ ParseCommandLine (): parser = argparse.ArgumentParser (description='Fix cfs bug.') Parser.add_argument ('- dry_run', dest='dry_run', action='store_true', help='Whether or not to run in dry_run mode') parser.set_defaults (dry_run=False) parser.add_argument ('- sys', type=str, default='/sys', help='The root directory of the / sys fs') parser.add_argument ('- interval', type=int) Help='Seconds to wait between invocations') return parser.parse_args () def ReadFile (directory, filename): with open (os.path.join (directory, filename),'r') as f: return f.read () def WriteFile (directory, filename, number): with open (os.path.join (directory, filename) 'w') as f: f.write ('% d'% number) def ListSubdirs (directory): return [os.path.join (directory, f) for f in os.listdir (directory) if os.path.isdir (os.path.join (directory, f))] def CalculateNewQuotaPeriodForPod (quota Period): scaled_times = math.ceil ((math.log (period)-math.log (100000)) / (math.log-math.log (100000)) new_period = 100000 new_quota = math.floor (quota * math.pow (128.0 / 147, scaled_times)) + 1 + scaled_times return new_quota, new_perioddef CalculateNewQuotaPeriodForContainer (quota Period): scaled_times = math.ceil ((math.log (period)-math.log (100000)) / (math.log (147147-math.log)) new_period = 100000 new_quota = math.floor (quota * math.pow (128.0 / 147147, scaled_times)) new_quota = max (new_quota, 1000) return new_quota, new_perioddef FixPodIfAffected (pod_dir, dry_run): try: quota = long (ReadFile (pod_dir) 'cpu.cfs_quota_us')) period = long (ReadFile (pod_dir,' cpu.cfs_period_us')) except IOError: print ('Skipping pod_dir% s. The pod may have disappeared from cfs before',' it could be examined'% (pod_dir)) return if quota

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.