Shulou(Shulou.com)05/31 Report--

In this article, the editor introduces in detail "what is the stability framework of Kubernetes", the content is detailed, the steps are clear, and the details are handled properly. I hope this article "what is the stability framework of Kubernetes" can help you solve your doubts.

Introduction

Using Kubernetes version 1.15, when querying Kubernetes monitoring metrics through API (/ metrics), words such as "[ALPHA]", "[STABLE]" and "(Deprecated)" can be seen in the HELP information of some metrics to indicate the stability of monitoring metrics, so that users can safely use these metrics to monitor clusters according to their own needs.

A typical indicator is as follows:

# HELP rest_client_request_latency_seconds [ALPHA] (Deprecated) Request latency in seconds. Broken down by verb and URL.# TYPE rest_client_request_latency_seconds histogramrest_client_request_latency_seconds_bucket {url= "https://[::1]:6443/%7Bprefix%7D",verb="GET",le="0.001"} 36rest_client_request_latency_seconds_bucket {url=" https://[::1]:6443/%7Bprefix%7D",verb="GET", Le= "0.002"} 148rest_client_request_latency_seconds_bucket {url= "https://[::1]:6443/%7Bprefix%7D",verb="GET",le="0.004"} 166rest_client_request_latency_seconds_bucket {url=" https://[::1]:6443/%7Bprefix%7D",verb="GET",le="0.008"} 172

Like other API logos, ALPHA means that the monitoring indicator is still being processed in the experimental stage and may change at any time, and there is no guarantee that users will be notified and that they need to be used with caution. Deprecated means that the monitoring metric has been deprecated and will be deleted in future versions, and users need to stop using or use alternative monitoring metrics as soon as possible.

This article was written on the eve of the release of version 1.16, and all the existing monitoring indicators have not yet been migrated to the framework, so only some of the monitoring indicators will have stability identification.

This paper hopes to start with the development background of the monitoring index stability framework, and then introduce the implementation principle of the framework to help developers understand and use this feature in more detail.

Background

Many cluster monitoring systems use the monitoring indicators provided by Kubernetes to monitor the operation of the cluster. However, the indicators provided by Kubernetes do not have any guarantee of stability, that is, Kubernetes may modify these indicators at any time. After users upgrade to the new version of Kubernetes, the original monitoring indicators may have been modified or deleted directly, which brings trouble to users, which has attracted the attention of the community.

In order to solve this problem, the community has conducted extensive discussions and summed up these two mainstream ideas:

Use version tags: classify version identifiers for monitoring metrics, such as / metrics/v1beta1, / metrics/v1, etc.

Use document tags: provide document tags with stable monitoring metrics

Both of these ideas have corresponding advantages and disadvantages:

The method of using version tags is more similar to that of Kubernetes API, which has the advantage of supporting multiple versions at the same time, but the disadvantage is that it is very cumbersome to implement.

The advantage of using document marking is that it is easy to achieve, and the stable monitoring indicators can be recorded in the document, but the disadvantage is that the rule will be inevitably broken in subsequent iterations, which is very difficult to manage from an engineering point of view.

Finally, combining the advantages and disadvantages of these two ideas, the community uses a more innovative scheme, that is, the stability framework to be introduced in this article, which is easier to use version tags at the implementation level and easier to manage at the engineering level than using document tags.

Vision

Just as every company needs a vision or slogan when it is founded, the stability framework for monitoring indicators has its own vision:

Provide a mechanism to describe the stability of monitoring indicators

Provide a variety of monitoring stability descriptions

There are two visions, one is to help Kubernetes developers define their own monitoring metrics, and the other is how to present stability assurance to users.

At the same time, the following are not considered by the framework:

Do not define a specific monitoring indicator stability level

Not responsible for checking whether the specific monitoring indicators are in line with the specifications.

In fact, these elements that are not taken into account in this framework are also important and will be done in other matters of the community, but the framework is not responsible for them.

The implementation of the old principle

We may all have heard that Prometheus has become the de facto standard for Kubernetes monitoring, which is reflected in the fact that many components of Kubernetes (kube-scheduler, kubelet, kube-controller-manager, kube-apiserver, etc.) use Prometheus clients to generate monitoring metrics in Prometheus format.

With Prometheus, an instance of each monitoring metric is generally initialized and then added to the global registry. Finally, when the user accesses it using API, http handler is responsible for exporting these monitoring metrics.

Initialize the instance:

Var authenticatedUserCounter = prometheus.NewCounterVec (prometheus.CounterOpts {Name: "authenticated_user_requests", Help: "Counter of authenticated requests broken out by username.",}, [] string {"username"},)

Add to the registry:

Prometheus.MustRegister (authenticatedUserCounter)

Http handler also uses the handler provided by Prometheus:

/ / Install adds the DefaultMetrics handlerfunc (m DefaultMetrics) Install (c * mux.PathRecorderMux) {register () c.Handle ("/ metrics", prometheus.Handler ())}

From the above, we can see that the original monitoring markers rely heavily on the characteristics of Prometheus, while the new framework design will encapsulate the definition, initialization and registration process of monitoring indicators, that is, each component will no longer directly use Prometheus, but use the new framework. Although the new framework is still implemented with Prometheus, it provides more personalized design.

New implementation: monitoring indication definition

In the part of monitoring index definition, the new framework provides a new index parameter structure in order to increase the stability identification.

For example, when you used Prometheus to define a counter, you would use the prometheus.CounterOpts structure, as follows:

Var someMetricDefinition = prometheus.CounterOpts {Name: "some_metric", Help: "some description",}

The new framework provides an extended parameter structure kubemetrics.CounterOpts, and new fields such as StabilityLevel and DeprecatedVersion are added to indicate stability and deprecation marks.

Indicates that a monitoring metric is discarded, which can be defined as follows:

Var deprecatedMetricDefinition = kubemetrics.CounterOpts {Name: "some_deprecated_metric", Help: "some description", StabilityLevel: kubemetrics.STABLE, / / Custom field, Stability ID DeprecatedVersion: "1.15", / / Custom field, deprecate ID}

Represents an ALPHA monitoring metric, which can be defined as follows:

Var alphaMetricDefinition = kubemetrics.CounterOpts {Name: "some_alpha_metric", Help: "some description", StabilityLevel: kubemetrics.ALPHA, DeprecatedVersion: "1.15", / / optional field indicating which version will be deprecated}

Consistent with Prometheus, the new framework extends four monitoring metrics:

CounterOpts

GaugeOpts

HistogramOpts

SummaryOpts

This section does not introduce each monitoring index for the time being, or focuses on the implementation of the new framework.

New implementation: monitoring indicator initialization

Once upon a time, using Prometheus to initialize monitoring metrics (to get monitoring metric instances) would use a series of prometheus.NewXXX () methods, as follows:

Var someCounterVecMetric = prometheus.NewCounterVec (someMetricDefinition, / / Metric definition [] string {"some-label", "other-label"}, / / Metric label}

On the one hand, because the new framework extends the indicator definition structure, it can no longer use the prometheus.NewXXX () method, on the other hand, the new framework also hopes to extend the custom behavior when the indicator is initialized (such as dealing with the fields added by the definition link), so a series of similar methods are provided in the new framework.

An example of initializing an example using the new framework is as follows:

Var deprecatedMetric = kubemetrics.NewCounterVec (deprecatedMetricDefinition, / / extended metric definition [] string {"some-label", "other-label"},}

Also consistent with Prometheus, the new framework also provides all the initialization methods:

NewCounter () and NewCounterVec ()

NewGauge () and NewGaugeVec ()

NewHistogram () and NewHistogramVec ()

NewSummary () and NewSummaryVec ()

Similarly, this section does not introduce each initialization method for the time being, which is left to later chapters.

New implementation: monitoring indicator registration

Once upon a time, when using Prometheus to register an instance of monitoring metrics, it was actually registered to a global registry, as shown below:

Prometheus.MustRegister (someCounterVecMetric)

Registration is also encapsulated in the new framework to add custom behaviors, such as obsolete version checking. The pseudo code of the registration logic is as follows:

Import version "k8s.io/apimachinery/pkg/version" type Registry struct {promregistry * prometheus.Registry KubeVersion version.Info} func (r * Registry) MustRegister (metric kubemetrics.Metric) {if metricutils.compare (metric.DeprecatedVersion) .isLessThan (r.KubeVersion) {/ / if the obsolete version is lower than the current version, check for obsolete rules Do not register / / check if binary has deprecated metrics enabled otherwise / / no-op registration return} else if metricutils.compare (metric.DeprecatedVersion) .isEqual (r.KubeVersion) {/ / if the obsolete version is consistent with the current version Then add the deprecation information to the HELP information of the indicator and record the warning log / / append deprecated text to description / / emit warning in logs / / continue to actual registration} / / if it is an ALPHA monitoring indicator, add the corresponding identification to the HELP information r.promregistry.MustRegister (metric.realMetric) / / finally use Prometheus to complete the registration}

Register with the new framework, as follows:

Kubemetrics.MustRegister (deprecatedMetric) kubemetrics.MustRegister (alphaMetric)

From the above summary, we can see that at the programming level, the existing monitoring indicators can be easily migrated to the new framework. If the monitoring indicators do not have personalized requirements, then their behavior is completely consistent with the original Prometheus. If there are personalized requirements, they can be completed through simple configuration, and the new framework is responsible for all the specific implementation details.

Stability statement

The current version (kubernetes 1.15-1.16) provides two levels of stability to illustrate the stability of each monitoring metric:

Alpha

Stable

When additional mention is needed, the Beta level is not taken into account in the current design, and the authors of the new framework say that it will be very easy to add it in the future if necessary.

Alpha-level monitoring indicators basically do not provide any guarantee of stability, they can be modified at any time, and after the introduction of the new framework, the existing monitoring indicators will be marked for this level.

Stable-level monitoring instructions are basically guaranteed not to be modified unless they are discarded in the future. The term "no longer modified" here means that the following three contents will not be modified:

Monitoring metrics themselves will not be deleted or renamed

The type of monitoring indicator will not be modified

The lable list of monitoring metrics will not increase or decrease

For Stable-level monitoring metrics, the lable list does not change, but the lable value can be changed. For example, a metric is used to record the number of authentication, and use "result" as the lable to distinguish. The original lable value is "success" and "failure", which allows you to add a new lable value, such as adding a "error" (not a simple authentication failure, but some exception).

For example, the indicators you got before will be:

Authentication_attempts {result= "success"} 1345authentication_attempts {result= "failure"} 100

Become:

Authentication_attempts {result= "success"} 1345authentication_attempts {result= "failure"} 100authentication_attempts {result= "error"} 1 / / add a new lable value

Generally speaking, this change will not have a great impact on users.

For Stable-level monitoring metrics, deleting or adding lable is not allowed. If you have to do so, you need to discard the current metrics and then provide a new monitoring metric.

As far as users are concerned, the new framework also provides an explicit ability for users to block certain monitoring metrics. By default, all monitoring metrics are registered and eventually provided to users through API, but after using the new framework, users can explicitly turn off some monitoring metrics through the startup parameters of the corresponding components, such as:

-- disable-metrics=somebrokenmetric,anothermetric stability guarantee

Marking a monitoring indicator as Stable means making a commitment to a large number of users, which is consistent with API. Therefore, when marking a monitoring indicator as Stable or abandoning a monitoring indicator, you need to be very careful with review.

However, according to the organizational division of the Kubernetes community, each component has different group (accurately called SIG) to be responsible for the corresponding component development, and the reviewer of each group may not fully understand the concept introduced by the new framework, so the group is likely to break the stability commitment made before. However, the modification of each relevant monitoring index is review by the group responsible for monitoring, which will be very expensive and become undesirable.

To address this problem, the kubernetes community offers an eye-catching way to introduce new conformance testing. That is, generate an existing stability list, store it in a directory managed by the monitoring group, and add a new CI workflow to collect the latest stability monitoring metrics. If the two are inconsistent, CI will fail and refuse to integrate, unless the stability list is explicitly modified. CI ensures that when explicitly modifying the stability list, it must be approved by the monitoring group.

Abandonment rule

After the monitoring metrics of each Stable are decided to be discarded, you can explicitly specify the version to be deprecated (DeprecatedVersion) at the monitoring metrics definition. After a metric is marked as obsolete, it will not be deleted immediately. It needs to be left to the user with an appropriate window, which will only be deleted in some subsequent versions.

Abandoned monitoring metrics will go through the following stages (each phase represents a kubernetes minor version):

Stable metric-> Deprecated metric-> Hidden metric-> Deletion

For example, if a monitoring metric enters the Stable phase in version 1.16 and will be deprecated in version 1.17, its status in each version is as follows:

1.16 (Stable metric): can be used normally

1.17 (Deprecated metric): marked as deprecated, the current version is still available, but the user needs to prepare for adaptation

1.18 (Hidden metric): the monitoring metric is not enabled by default, and the administrator can explicitly enable it through the parameter.

1.19 (Deletion): completely deleted and can no longer be used

Abandonment stage

For example, if a monitoring metric will be obsolete in version 1.17, you can set it like this when developing version 1.17 (or earlier):

Var someCounter = kubemetrics.CounterOpts {Name: "some_counter", Help: "this counts things", StabilityLevel: kubemetrics.STABLE, DeprecatedVersion: "1.17", / / this metric is deprecated when the Kubernetes version = = 1.17}

Using versions prior to 1.17, the monitoring metric information is as follows:

# HELP some_counter this counts things# TYPE some_counter countersome_counter 0

Then when you use version 1.17, you will see the deprecation information in the corresponding metric information:

# HELP some_counter (Deprecated from 1.17) this counts things# TYPE some_counter countersome_counter 0

In addition, when the monitoring indicator is marked as obsolete, although it can be used normally, the alarm log can be seen in the corresponding component log.

Hidden phase

When the next version of the monitoring metric is deprecated, that is, DeprecatedVersion = = current_kubernetes_version-1, the monitoring metric will be hidden by default: that is, it will not be automatically registered.

Hidden metrics can be explicitly enabled through the startup parameter (--enable-hidden-metrics=really_deprecated_metric) of the corresponding component.

If users forget to adapt in the previous version, they can still use it in this version, which undoubtedly extends the adaptation window for users.

In addition, several points that need to be mentioned in particular are:

The monitoring indicators marked as deprecated still respect the previous stability agreement and will not modify the content of the indicator except by adding the deprecation mark.

This discarding rule is for Stable monitoring metrics and does not impose mandatory requirements on Alpha monitoring metrics.

Monitoring metrics in the Alpha phase can also be marked with obsolete version numbers, which can help users using Alpha monitoring metrics to know exactly when a monitoring metric will be deleted. It is only used to indicate that you do not have to follow the abandonment rules before being deleted.

After reading this, the article "what is the stability framework of Kubernetes" has been introduced. If you want to master the knowledge points of this article, you still need to practice and use it yourself. If you want to know more about related articles, welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.