Shulou(Shulou.com)06/01 Report--

This article mainly introduces "how to customize Istio indicators to achieve Pod horizontal automatic scaling and automatic scaling Spring Boot". In daily operation, I believe many people have doubts about how to customize Istio indicators to achieve Pod horizontal automatic scaling and automatic scaling Spring Boot. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. Hope to help you answer the question of "how to customize Istio indicators to achieve Pod horizontal automatic scaling and automatic scaling Spring Boot"! Next, please follow the editor to study!

One of the core functions of Pipeline, the Banzai Cloud application and the devops container management platform, is multidimensional and can be automatically adjusted based on default and custom metrics. After we introduced custom metrics, we chose to collect metrics from Prometheus through the Prometheus adapter. Since then, many of our customers have started using Hoizontal Pod Autoscaling, and most of them are only satisfied with basic CPU and memory metrics.

We have always known that this is not an ideal solution, and we have been trying to find more flexible solutions in order to:

Extension based on custom Prometheus metrics

Provide extended support for more complex Prometheus queries

With the release of our open source Istio operator and the widespread introduction of the Istio-based service grid on the Pipeline platform, we also provide automatic scaling according to custom Istio metrics. Prometheus now provides network metrics, latency, failure rate, and more (removed from Istio by default). Our HPA operator decides whether to expand or not based on these metrics.

However, when using Prometheus Adapter, our approach is no longer feasible, so we decided to explore alternatives.

If you miss our series of blogs about the different types of auto-scaling features supported and automated by the Pipeline platform, please check out these earlier posts:

Autoscaling Kubernetes cluster

Vertical pod autoscaler

Automatically scale Kubernetes deployment horizontally on custom metrics

Horizontal Pod Autoscaler Kubernetes Operator

Reload horizontal Pod automatic regulator operator

TL;DR

Introduction to redesigned custom metrics support in Pipeline and its HPA Operator.

An example of automatically adjusting a Spring Boot deployment using Istio metrics from Prometheus

Learn more about what happens after adding custom metrics

Switch to another custom metric adapter, introduction to kube-metrics-adapter

Re-support custom metrics

For the reasons listed above, we decided to use another custom metric adapter, kube-metrics-adapter.

Kube-metrics-adapter is a common metrics adapter that collects and provides metrics from multiple sources. For each source, there is a Collector implementation; currently, we are most interested in its Prometheus collector.

Prometheus Collector is a general-purpose collector. It maps Prometheus queries to metrics that can be used by the HPA controller to deploy automatic extensions. Its approach is different from Prometheus Adapter in that it has a predefined set of rules-containing queries run against Prometheus for converting metrics to custom metrics-executed on a regular basis. Get all matching metrics criteria (because these queries must be generic and locate all pod and deployments) and convert and expose them as custom metrics. In contrast, "Prometheus collector" collects only the metric results returned by a custom query, its definition varies from deployment to deployment, and only for a single deployment / pod or service, thus reducing the total number of metrics stored. One drawback of this solution is that currently, users should avoid executing underperforming queries.

Use ISTIO to automatically scale Spring Boot applications

I chose the popular spotguides, MySQL-based Sprint Boot application to quickly launch a lightweight Web application container next to the MySQL server. I use Banzai Pipeline Beta, which supports rapid startup of Kubernetes clusters among six cloud providers or locally. Prometheus & Grafana Monitoring and Istio are default features that can be easily turned on with the UI switch (Pipeline handles the rest). In this example, I will start a Kubernetes cluster on Google Cloud, open Monitoring and Service Mesh, and deploy Spring Boot Spotguide on it. Cluster creation is part of our spotguide wizard, but because Service Mesh is a relatively new feature in Pipeline, we cannot (so far) enable it from the spotguide process, so we must create the cluster in advance.

I recorded these steps in this video:

Once your Spring Boot container and MySQL server are up and running, you can open the demo program from the Spotguid overview. This is a demo application, which is a typical Spring Web application. We have created a GitHub code base where you can find the hint code (just like our other spotguides). By default, the application URL will only return the health check status, which connects to the Kubernetes activity and readiness check. The JVM metrics will then be exposed to Prometheus, and repo will be integrated into our CI/CD stream, so once you submit the business logic, it will be automatically deployed to the cluster. Read more about our spotguides here.

You can download the Kubernetes configuration from the cluster list or the details page so that you can later obtain kubectl access to the cluster from the local computer.

I am using the hey tool (go get-u github.com/rakyll/hey) to generate the load for the Spring application; the following command will issue 50 requests per second, for a total of 10000 requests:

Hey-n 10000-Q 50 https://spring-boot-custom-metrics-demo.sancyx5g25.sancyx.beta.banzaicloud.io/actuator/health/kubernetes

Now, turn on Monitoring (the link provided in our Spotguide summary or cluster details page) to view the available metrics and determine your Prometheus query. Because we enabled Service Mesh, all network traffic will send metrics to the Istio telemetry service through the Envoy agent, which is grabbed by Prometheus.

Based on my example, I will query based on the istio_requests_total metric, which is related to the Spring container:

Sum (rate (istio_requests_total {destination_service= "spring-boot-custom-metrics-demo-spotguide-spring-boot.default.svc.cluster.local", destination_service_name= "spring-boot-custom-metrics-demo-spotguide-spring-boot", destination_service_namespace= "default", destination_workload= "spring-boot-custom-metrics-demo-spotguide-spring-boot"} [1m]))

Be sure to add appropriate label filters to specifically select metrics relevant to your pod/ deployment.

Select spring-boot-custom-metrics-demo-spotguide-spring-boot deployment from the Horizontal Pod Autoscaler menu on the deployment list page to access the HPA Edit page. There, you can set cpu, memory, and custom metrics by entering the name of the custom metric or query.

Now, if you generate another load (50 requests per second) over a longer period of time, you should increase the replica count in the HPA that belongs to the deployment:

Hey-n 50000-Q 50 https://spring-boot-custom-metrics-demo.sancyx5g25.sancyx.beta.banzaicloud.io/actuator/health/kubernetes...kubectl get hpaNAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGEspring-boot-custom-metrics-demo-spotguide-spring-boot Deployment/spring -boot-custom-metrics-demo-spotguide-spring-boot 245866m/40 1 10 7 80s

After loading, the number of copies is slowly reduced to the default minimum:

The scene behind kubectl get hpaNAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGEspring-boot-custom-metrics-demo-spotguide-spring-boot Deployment/spring-boot-custom-metrics-demo-spotguide-spring-boot 266m/40 1 10 1 32m

Let's take a closer look at what happens behind the scenes when you use Pipeline UI to add custom metrics to your deployment. This process is very similar to resource metrics, which you may be familiar with from a series of previous posts, but for now, let's focus on API related to custom metrics.

Pipeline UI uses HPA API to create / update metrics:

PUT {{pipeline_url}} / api/v1/orgs/:orgId/clusters/:clusterId/hpa {"scaleTarget": "example-deployment-name", "minReplicas": 1, "maxReplicas": 10, "customMetrics": {"customMetricName": {"query": "sum ({kubernetes_pod_name=~\" ^ example-deployment-name-pod.*\ ", _ _ name__=~\" example-metric-name\ "})" "targetAverageValue": "100"}

If you want to specify an absolute value, you can use targetValue instead of targetAverageValue. The difference is that targetAverageValue averages the metrics obtained using the current pod copy count.

What Pipeline actually does is add the following comments to your deployment:

Hpa.autoscaling.banzaicloud.io/maxReplicas=10 hpa.autoscaling.banzaicloud.io/minReplicas=1 prometheus.customMetricName.hpa.autoscaling.banzaicloud.io/query=sum ({kubernetes_pod_name=~\ "^ example-deployment-name-pod.*\", _ _ name__=~\ "example-metric-name\"})

All of this is done through HPA operator, which manages HPA resources based on deployment annotations. Pipeline deploys HPA operator through a post hook after the cluster is created, and then deploys kube-metrics-adapter and metrics-server if metrics.api is not already registered. This process may vary, depending on your cloud provider and K8s version, which may already have some features installed. In our example, the newly created HPA object will look like this:

ApiVersion: v1itemsHorizontalPodAutoscaler metadata-apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata: annotations:... Autoscaling.alpha.kubernetes.io/current-metrics:'[{"type": "Object", "object": {"target": {"kind": "Pod", "name": "example-deployment-name-customMetricName", "apiVersion": "v1"}, "metricName": "customMetricName", "currentValue": "222m"}}] 'autoscaling.alpha.kubernetes.io/metrics:' [{"type": "Object", "object": {"target": {"kind": "Pod" "name": "example-deployment-name-customMetricName", "apiVersion": "v1"}, "metricName": "customMetricName", "targetValue": "40"}] 'metric-config.object.customMetricName.prometheus/per-replica: "true" metric-config.object.customMetricName.prometheus/query: sum ({kubernetes_pod_name=~\ "^ example-deployment-name-pod.*\", _ _ name__=~\ "example-metric-name\"}). Spec: maxReplicas: 10 minReplicas: 1 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: example-deployment-name status: currentReplicas: 1 desiredReplicas: 1kind: Listmetadata: resourceVersion: "" selfLink: ""

Note that the custom metric is of type Object and is bound to a Pod resource named example-deployment-name-customMetricName. Kube-metrics-adapter also uses comment queries to get the measure, which exposes / apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/example-deployment-name-customMetricName/customMetricName at the following endpoints, and it is exactly where the HPA controller will look:

$kubectl get-- raw "/ apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/customMetricName" | jq. {"kind": "MetricValueList", "apiVersion": "custom.metrics.k8s.io/v1beta1", "metadata": {"selfLink": "/ apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/customMetricName"} "items": [{"describedObject": {"kind": "Pod", "namespace": "default", "name": "example-deployment-name-customMetricName", "apiVersion": "v1"}, "metricName": "customMetricName", "timestamp": "2019-03-13T20:23:32Z" "value": "222m"}]} so far On "how to customize Istio indicators to achieve Pod horizontal automatic scaling and automatic scaling Spring Boot" study is over, I hope to be able to solve everyone's doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.