Shulou(Shulou.com)05/31 Report--

This article mainly explains "how to use Kubernetes HPA Controller". The content in the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn how to use Kubernetes HPA Controller.

Analysis of Source Code Directory structure

The main code of HorizontalPodAutoscaler (hereinafter referred to as HPA) is as follows, and there are not many files involved.

Startup code of cmd/kube-controller-manager/app/autoscaling.go / / HPA Controller / pkg/controller/podautoscaler. Core code of ├── BUILD ├── OWNERS ├── doc.go ├── horizontal.go / / podautoscaler Include the creation of ├── horizontal_test.go ├── metrics │ ├── BUILD │ ├── metrics_client.go │ ├── metrics_client_test.go │ metrics_client_test.go.orig │ ├── metrics_client_test.go.rej │ └── utilization.go ├── replica_calculator.go / / ReplicaCaculator and the method of calculating replicas according to cpu/metrics replica_calculator_test.go

Horizontal.go and replica_calculator.go are the core files, and their corresponding Structure is as follows:

Horizontal.go

Replica_calculator.go

Source code analysis

HPA Controller, like other Controller, is initialized and started when kube-controller-manager starts, as shown in the following code.

Cmd/kube-controller-manager/app/controllermanager.go:224func newControllerInitializers () map [string] InitFunc {controllers: = map [string] InitFunc {}... Controllers ["horizontalpodautoscaling"] = startHPAController... Return controllers}

When kube-controller-manager starts, it initial a bunch of controllers. For HPA controller, its startup is left to startHPAController.

Cmd/kube-controller-manager/app/autoscaling.go:29func startHPAController (ctx ControllerContext) (bool, error) {. / / HPAController requires that the cluster has deployed Heapster, and Heapster provides monitoring data for replicas calculation. MetricsClient: = metrics.NewHeapsterMetricsClient (hpaClient, metrics.DefaultHeapsterNamespace, metrics.DefaultHeapsterScheme, metrics.DefaultHeapsterService, metrics.DefaultHeapsterPort,) / / create the ReplicaCaculator, which will be used to calculate the desired replicas later. ReplicaCalc: = podautoscaler.NewReplicaCalculator (metricsClient, hpaClient.Core ()) / / create the HPA Controller and start goroutine to execute its Run method to get to work. Go podautoscaler.NewHorizontalController (hpaClient.Core (), hpaClient.Extensions (), hpaClient.Autoscaling (), replicaCalc, ctx.Options.HorizontalPodAutoscalerSyncPeriod.Duration,) .Run (ctx.Stop) return true, nil}

First, let's take a look at the code that NewHorizontalController creates HPA Controller.

Pkg/controller/podautoscaler/horizontal.go:112func NewHorizontalController (evtNamespacer v1core.EventsGetter, scaleNamespacer unversionedextensions.ScalesGetter, hpaNamespacer unversionedautoscaling.HorizontalPodAutoscalersGetter, replicaCalc * ReplicaCalculator, resyncPeriod time.Duration) * HorizontalController {... / / build HPA Controller controller: = & HorizontalController {replicaCalc: replicaCalc, eventRecorder: recorder, scaleNamespacer: scaleNamespacer, hpaNamespacer: hpaNamespacer } / / create Informer Configure the corresponding ListWatch Func and its corresponding EventHandler to monitor the Add and Update events of the HPA Resource. NewInformer is the core code entry for HPA. Store, frameworkController: = newInformer (controller, resyncPeriod) controller.store = store controller.controller = frameworkController return controller}

It is necessary to look at the definition of HPA Controller struct:

Pkg/controller/podautoscaler/horizontal.go:59type HorizontalController struct {scaleNamespacer unversionedextensions.ScalesGetter hpaNamespacer unversionedautoscaling.HorizontalPodAutoscalersGetter replicaCalc * ReplicaCalculator eventRecorder record.EventRecorder / / A store of HPA objects, populated by the controller. Store cache.Store / / Watches changes to all HPA objects. Controller * cache.Controller}

ScaleNamespacer is actually a ScaleInterface, including Scale subresource's Get and Update interfaces.

HpaNamespacer is HorizontalPodAutoscalerInterface, including Create, Update, UpdateStatus, Delete, Get, List, Watch and other interfaces of HorizontalPodAutoscaler.

ReplicaCalc calculates the corresponding desired replicas based on the monitoring data provided by Heapster.

Pkg/controller/podautoscaler/replica_calculator.go:31 type ReplicaCalculator struct {metricsClient metricsclient.MetricsClient podsGetter v1core.PodsGetter}

Store and controller:controller are used for watch HPA objects and updated to the store cache.

Scale subresource is mentioned above. What is that? Well, we need to look at the definition of Scale.

Pkg/apis/extensions/v1beta1/types.go:56// represents a scaling request for a resource.type Scale struct {metav1.TypeMeta `json: ", inline" `/ / Standard object metadata; More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#metadata. / / + optional v1.ObjectMeta `json: "metadata,omitempty" protobuf: "bytes,1,opt,name=metadata" `/ / defines the behavior of the scale. More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#spec-and-status. / / + optional Spec ScaleSpec `json: "spec,omitempty" protobuf: "bytes,2,opt,name=spec" `/ / current status of the scale. More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#spec-and-status. Read-only. / / + optional Status ScaleStatus `json: "status,omitempty" protobuf: "bytes,3,opt,name=status" `} / / describes the attributes of a scale subresourcetype ScaleSpec struct {/ / desired number of instances for the scaled object. Replicas int `json: "replicas,omitempty" `} / / represents the current status of a scale subresource.type ScaleStatus struct {/ / actual number of observed instances of the scaled object. Replicas int `json: "replicas" `/ / label query over pods that should match the replicas count. Selector map [string] string `json: "selector,omitempty" `}

Scale struct serves as the request data for a scale action.

Where Spec defines desired replicas number.

ScaleStatus defines current replicas number.

After looking at the structure of HorizontalController, go on to look at the newInformer called in NewHorizontalController. In the comments above, I mentioned that newInformer is the core code entry for the entire HPA.

Pkg/controller/podautoscaler/horizontal.go:75func newInformer (controller * HorizontalController, resyncPeriod time.Duration) (cache.Store, * cache.Controller) {return cache.NewInformer (/ / configure ListFucn and WatchFunc for periodic List and watch HPA resource. & cache.ListWatch {ListFunc: func (options v1.ListOptions) (runtime.Object, error) {return controller.hpaNamespacer.HorizontalPodAutoscalers (v1.NamespaceAll) .list (options)}, WatchFunc: func (options v1.ListOptions) (watch.Interface Error) {return controller.hpaNamespacer.HorizontalPodAutoscalers (v1.NamespaceAll) .Watch (options)},}, / define the object expected to be received as HorizontalPodAutoscaler & autoscaling.HorizontalPodAutoscaler {} / / define cycle resyncPeriod of periodic List, / / configure Handler of HPA resource event (AddFunc) UpdateFunc) cache.ResourceEventHandlerFuncs {AddFunc: func (obj interface {}) {hpa: = obj. (* autoscaling.HorizontalPodAutoscaler) hasCPUPolicy: = hpa.Spec.TargetCPUUtilizationPercentage! = nil _ HasCustomMetricsPolicy: = hpa.Annotations [HpaCustomMetricsTargetAnnotationName] if! hasCPUPolicy & &! hasCustomMetricsPolicy {controller.eventRecorder.Event (hpa, v1.EventTypeNormal, "DefaultPolicy", "No scaling policy specified-will use default one. See documentation for details ")} / / adjust hpa data according to monitoring err: = controller.reconcileAutoscaler (hpa) if err! = nil { Glog.Warningf ("Failed to reconcile% s:% v" Hpa.Name, err)}}, UpdateFunc: func (old Cur interface {}) {hpa: = cur. (* autoscaling.HorizontalPodAutoscaler) / / adjust the data of hpa according to monitoring err: = controller.reconcileAutoscaler (hpa) If err! = nil {glog.Warningf ("Failed to reconcile% s:% v" Hpa.Name, err)}}, / / We are not interested in deletions. },)}

The code of newInformer is not long, to put it simply, it is the Func of ListWatch configured with HPA resource, the Add of registered HPA resource and the handler Func of Update Event.

Finally, the data of hpa is corrected by calling reconcileAutoscaler.

In the above code, register the ListWatch Func of HPA resource as the List and Watch interfaces defined by HorizontalPodAutoscaler Interface.

Wait, having said so much, why haven't you seen the definition of HorizontalPodAutoscaler struct? All right, let's take a look, which happens to be in HorizontalPodAutoscaler Interface.

Pkg/apis/autoscaling/v1/types.go:76// configuration of a horizontal pod autoscaler.type HorizontalPodAutoscaler struct {metav1.TypeMeta `json: ", inline" `/ / Standard object metadata. More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#metadata / + optional v1.ObjectMeta `json: "metadata,omitempty" protobuf: "bytes,1,opt,name=metadata" `/ / behaviour of autoscaler. More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#spec-and-status. / / + optional Spec HorizontalPodAutoscalerSpec `json: "spec,omitempty" protobuf: "bytes,2,opt,name=spec" `/ / current information about the autoscaler. / / + optional Status HorizontalPodAutoscalerStatus `json: "status,omitempty" protobuf: "bytes,3,opt,name=status" `}

Spec HorizontalPodAutoscalerSpec stores the description information of hpa, and the information of corresponding flag can be configured through kube-controller-manager. Including the target CPU utilization TargetCPUUtilizationPercentage in the form of the average percentage of all pods corresponding to the minimum number of copies MinReplicas and the maximum number of copies MaxReplicas,hpa.

Pkg/apis/autoscaling/v1/types.go:36 / / specification of a horizontal pod autoscaler. Type HorizontalPodAutoscalerSpec struct {/ / reference to scaled resource; horizontal pod autoscaler will learn the current resource consumption / / and will set the desired number of pods by using its Scale subresource. ScaleTargetRef CrossVersionObjectReference `json: "scaleTargetRef" protobuf: "bytes,1,opt,name=scaleTargetRef" `/ / lower limit for the number of pods that can be set by the autoscaler, default 1. / / + optional MinReplicas * int32 `json: "minReplicas,omitempty" protobuf: "varint,2,opt,name=minReplicas"` / / upper limit for the number of pods that can be set by the autoscaler; cannot be smaller than MinReplicas. MaxReplicas int32 `json: "maxReplicas" protobuf: "varint,3,opt,name=maxReplicas" `/ / target average CPU utilization (represented as a percentage of requested CPU) over all the pods; / / if not specified the default autoscaling policy will be used. / / + optional TargetCPUUtilizationPercentage * int32 `json: "targetCPUUtilizationPercentage,omitempty" protobuf: "varint,4,opt,name=targetCPUUtilizationPercentage" `}

Status HorizontalPodAutoscalerStatu stores the current state data of HPA, including the time interval ObservedGeneration of the two scale, the timestamp LastScaleTime of the last scale, the current number of copies CurrentReplicas, and the current CPU utilization in the form of the average percentage of all pods corresponding to the expected number of copies DesiredReplicas,hpa.

Pkg/apis/autoscaling/v1/types.go:52 / / current status of a horizontal pod autoscaler type HorizontalPodAutoscalerStatus struct {/ / most recent generation observed by this autoscaler. / / + optional ObservedGeneration * int64 `json: "observedGeneration,omitempty" protobuf: "varint,1,opt,name=observedGeneration" `/ / last time the HorizontalPodAutoscaler scaled the number of pods; / / used by the autoscaler to control how often the number of pods is changed. / / + optional LastScaleTime * metav1.Time `json: "lastScaleTime,omitempty" protobuf: "bytes,2,opt,name=lastScaleTime" `/ / current number of replicas of pods managed by this autoscaler. CurrentReplicas int32 `json: "currentReplicas" protobuf: "varint,3,opt,name=currentReplicas" `/ / desired number of replicas of pods managed by this autoscaler. DesiredReplicas int32 `json: "desiredReplicas" protobuf: "varint,4,opt,name=desiredReplicas" `/ / current average CPU utilization over all pods, represented as a percentage of requested CPU, / / e.g. 70 means that an average pod is using now 70 of its requested CPU. / / + optional CurrentCPUUtilizationPercentage * int32 `json: "currentCPUUtilizationPercentage,omitempty" protobuf: "varint,5,opt,name=currentCPUUtilizationPercentage" `}

The code of newInformer shows that no matter whether the event of hpa resource is Add or update, reconcileAutoscaler is finally called to trigger the update of HorizontalPodAutoscaler data.

Pkg/controller/podautoscaler/horizontal.go:272func (a * HorizontalController) reconcileAutoscaler (hpa * autoscaling.HorizontalPodAutoscaler) error {... / / obtain the scale subresource data of the corresponding resource. Scale, err: = a.scaleNamespacer.Scales (hpa.Namespace) .Get (hpa.Spec.ScaleTargetRef.Kind Hpa.Spec.ScaleTargetRef.Name). / / get the current number of copies currentReplicas: = scale.Status.Replicas cpuDesiredReplicas: = int32 (0) cpuCurrentUtilization: = new (int32) cpuTimestamp: = time.Time {} cmDesiredReplicas: = int32 (0) cmMetric: = "" cmStatus: = "" cmTimestamp: = time. Time {} desiredReplicas: = int32 (0) rescaleReason: = "" timestamp: = time.Now () rescale: = true / / if the expected number of copies is 0 There is no scale operation. If scale.Spec.Replicas = = 0 {/ / Autoscaling is disabled for this resource desiredReplicas = 0 rescale = false} / / the expected number of copies cannot exceed the maximum number of copies configured in hpa else if currentReplicas > hpa.Spec.MaxReplicas {rescaleReason = "Current number of replicas above Spec.MaxReplicas" DesiredReplicas = hpa.Spec.MaxReplicas} / / the expected number of copies cannot be lower than the configured minimum number of copies else if hpa.Spec.MinReplicas! = nil & & currentReplicas

< *hpa.Spec.MinReplicas { rescaleReason = "Current number of replicas below Spec.MinReplicas" desiredReplicas = *hpa.Spec.MinReplicas } // 期望副本数最少为1 else if currentReplicas == 0 { rescaleReason = "Current number of replicas must be greater than 0" desiredReplicas = 1 } // 如果当前副本数在Min和Max之间，则需要根据cpu或者custom metrics（如果加了对应的Annotation）数据进行算法计算得到期望副本数。 else { // All basic scenarios covered, the state should be sane, lets use metrics. cmAnnotation, cmAnnotationFound := hpa.Annotations[HpaCustomMetricsTargetAnnotationName] if hpa.Spec.TargetCPUUtilizationPercentage != nil || !cmAnnotationFound { // 根据cpu利用率计算期望副本数 cpuDesiredReplicas, cpuCurrentUtilization, cpuTimestamp, err = a.computeReplicasForCPUUtilization(hpa, scale) if err != nil { // 更新hpa的当前副本数 a.updateCurrentReplicasInStatus(hpa, currentReplicas) return fmt.Errorf("failed to compute desired number of replicas based on CPU utilization for %s: %v", reference, err) } } if cmAnnotationFound { // 根据custom metrics数据计算期望副本数 cmDesiredReplicas, cmMetric, cmStatus, cmTimestamp, err = a.computeReplicasForCustomMetrics(hpa, scale, cmAnnotation) if err != nil { // 更新hpa的当前副本数 a.updateCurrentReplicasInStatus(hpa, currentReplicas) return fmt.Errorf("failed to compute desired number of replicas based on Custom Metrics for %s: %v", reference, err) } } // 取cpu和custom metric得到的期望副本数的最大值作为最终的desired replicas，并且要在min和max范围内。 rescaleMetric := "" if cpuDesiredReplicas >

DesiredReplicas {desiredReplicas = cpuDesiredReplicas timestamp = cpuTimestamp rescaleMetric = "CPU utilization"} if cmDesiredReplicas > desiredReplicas {desiredReplicas = cmDesiredReplicas timestamp = cmTimestamp rescaleMetric = cmMetric } if desiredReplicas > currentReplicas {rescaleReason = fmt.Sprintf ("% s above target" RescaleMetric)} if desiredReplicas

< currentReplicas { rescaleReason = "All metrics below target" } if hpa.Spec.MinReplicas != nil && desiredReplicas < *hpa.Spec.MinReplicas { desiredReplicas = *hpa.Spec.MinReplicas } // never scale down to 0, reserved for disabling autoscaling if desiredReplicas == 0 { desiredReplicas = 1 } if desiredReplicas >

Hpa.Spec.MaxReplicas {desiredReplicas = hpa.Spec.MaxReplicas} / / Do not upscale too much to prevent incorrect rapid increase of the number of master replicas caused by / / bogus CPU usage report from heapster/kubelet (like in issue # 32304). If desiredReplicas > calculateScaleUpLimit (currentReplicas) {desiredReplicas = calculateScaleUpLimit (currentReplicas)} / / according to the comparison between currentReplicas and desiredReplicas, and whether the scale time meets the configuration interval requirements Decide whether rescale rescale = shouldScale (hpa, currentReplicas, desiredReplicas, timestamp)} if rescale {scale.Spec.Replicas = desiredReplicas / / executes the Update interface of ScaleInterface at this time, triggering the data update of the corresponding resource scale subresource that calls API Server. In fact, the replicas corresponding to rc or deployment will eventually be modified, and then rc or deployment Controller will eventually expand or reduce the capacity to make the number of copies reach the new expected value. _, err = a.scaleNamespacer.Scales (hpa.Namespace) .Update (hpa.Spec.ScaleTargetRef.Kind, scale) if err! = nil {a.eventRecorder.Eventf (hpa, v1.EventTypeWarning, "FailedRescale", "New size:% d; reason:% s" Error:% v ", desiredReplicas, rescaleReason, err.Error () return fmt.Errorf (" failed to rescale% s:% v ", reference, err)} a.eventRecorder.Eventf (hpa, v1.EventTypeNormal," SuccessfulRescale "," New size:% d " Reason:% s ", desiredReplicas, rescaleReason) glog.Infof (" Successfull rescale of% s, old size:% d, new size:% d, reason:% s ", hpa.Name, currentReplicas, desiredReplicas, rescaleReason)} else {desiredReplicas = currentReplicas} / / Update status data return a.updateStatus (hpa, currentReplicas, desiredReplicas, cpuCurrentUtilization, cmStatus) of hpa resource Rescale)}

The above reconcileAutoscaler code is very important, writing everything you want to say to the corresponding comments. ComputeReplicasForCPUUtilization and computeReplicasForCustomMetrics need to be mentioned separately, because these two methods are the embodiment of the HPA algorithm. In fact, the final algorithm is implemented in pkg/controller/podautoscaler/replica_calculator.go:45#GetResourceReplicas and pkg/controller/podautoscaler/replica_calculator.go:153#GetMetricReplicas:

Pkg/controller/podautoscaler/replica_calculator.go:45#GetResourceReplicas is responsible for calculating the desired replicas number based on the cpu utilization data provided by heapster.

Pkg/controller/podautoscaler/replica_calculator.go:153#GetMetricReplicas is responsible for calculating the desired replicas number based on the custom raw metric data provided by heapster.

Specific about the HPA algorithm source code analysis, I will write a separate blog, interested can follow (for the vast majority of students do not need to pay attention to, unless you need to customize the HPA algorithm, will analyze specifically).

All in all, after the desired replicas is calculated from the cpu and custom metric data, take the maximum value of both, but not more than the configured Max Replicas.

Wait a moment. It is enough to calculate the desired replicas. We also need to use shouldScale to see whether the time interval between the current and the last auto scaling meets the conditions:

The interval between the two downsizing should not be less than 5min.

The interval between the two expansions must not be less than 3min.

The code for shouldScale is as follows:

Pkg/controller/podautoscaler/horizontal.go:387...var downscaleForbiddenWindow = 5 * time.Minutevar upscaleForbiddenWindow = 3 * time.Minute...func shouldScale (hpa * autoscaling.HorizontalPodAutoscaler, currentReplicas, desiredReplicas int32 Timestamp time.Time) bool {if desiredReplicas = = currentReplicas {return false} if hpa.Status.LastScaleTime = = nil {return true} / / Going down only if the usageRatio dropped significantly below the target / / and there was no rescaling in the last downscaleForbiddenWindow. If desiredReplicas

< currentReplicas && hpa.Status.LastScaleTime.Add(downscaleForbiddenWindow).Before(timestamp) { return true } // Going up only if the usage ratio increased significantly above the target // and there was no rescaling in the last upscaleForbiddenWindow. if desiredReplicas >

CurrentReplicas & & hpa.Status.LastScaleTime.Add (upscaleForbiddenWindow) .Before (timestamp) {return true} return false}

Only when this condition is met will the Scales.Update API be called to interact with the API Server to complete the setting of the replicas of the RC corresponding to the Scale. Take rc Controller as an example (similar to deployment Controller). The implementation logic of the Scales.Update API corresponding to API Server is as follows:

Pkg/registry/core/rest/storage_core.go:91func (c LegacyRESTStorageProvider) NewLegacyRESTStorage (restOptionsGetter generic.RESTOptionsGetter) (LegacyRESTStorage, genericapiserver.APIGroupInfo, error) {. If autoscalingGroupVersion: = (schema.GroupVersion {Group: "autoscaling", Version: "v1"}); registered.IsEnabledVersion (autoscalingGroupVersion) {apiGroupInfo.SubresourceGroupVersionKind ["replicationcontrollers/scale"] = autoscalingGroupVersion.WithKind ("Scale")}. RestStorageMap: = map [string] rest.Storage {... "replicationControllers": controllerStorage.Controller, "replicationControllers/status": controllerStorage.Status,...} return restStorage, apiGroupInfo, nil} pkg/registry/core/controller/etcd/etcd.go:124func (r * ScaleREST) Update (ctx api.Context, name string, objInfo rest.UpdatedObjectInfo) (runtime.Object, bool, error) {rc, err: = r.registry.GetController (ctx, name) & metav1.GetOptions {}) if err! = nil {return nil, false, errors.NewNotFound (autoscaling.Resource ("replicationcontrollers/scale"), name)} oldScale: = scaleFromRC (rc) obj, err: = objInfo.UpdatedObject (ctx, oldScale) if err! = nil {return nil, false, err} if obj = = nil {return nil, false Errors.NewBadRequest ("nil update passed to Scale")} scale, ok: = obj. (* autoscaling.Scale) if! ok {return nil, false, errors.NewBadRequest (fmt.Sprintf ("wrong object passed to Scale update:% v", obj))} if errs: = validation.ValidateScale (scale) Len (errs) > 0 {return nil, false, errors.NewInvalid (autoscaling.Kind ("Scale"), scale.Name, errs)} / / set rc to the expected number of replicas in Scale rc.Spec.Replicas = scale.Spec.Replicas rc.ResourceVersion = scale.ResourceVersion / / Update to etcd rc, err = r.registry.UpdateController (ctx Rc) if err! = nil {return nil, false, err} return scaleFromRC (rc), false, nil}

Students who know kubernetes rc Controller know very well that after modifying the replicas of rc, it will be rc Controller watch, and then trigger rc Controller to create or destroy the number of replicas with the corresponding difference, and finally make the number of copies reach the expected value calculated by HPA. In other words, it is up to rc controller to perform specific capacity expansion or reduction actions.

Finally, take a look at HorizontalController's Run method:

Pkg/controller/podautoscaler/horizontal.go:130func (a * HorizontalController) Run (stopCh

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.