Shulou(Shulou.com)05/31 Report--

The content of this article mainly focuses on how to manage the state of Kubernetes objects, the content of the article is clear and clear, it is very suitable for beginners to learn, it is worth reading. Interested friends can follow the editor to read together. I hope you can get something through this article!

Our main focus below is to explore how to manage the state of Kubernetes objects in a reliable and persistent way. As mentioned in previous articles, API Server itself is stateless, and it is the only component that can communicate directly with distributed storage etcd.

A brief description of etcd

In the * nix operating system, we usually use / etc to store relevant configuration data. In fact, the name etcd evolved from this, adding a "d" after etc to indicate "distributed" distributed. Any distributed system needs something like etcd that can store system data so that it can retrieve relevant data in a consistent and reliable manner. In order to achieve distributed data access, etcd uses the Raft protocol. Conceptually, the data model supported by etcd is key-value storage. In etcd2, each key exists in a hierarchical structure, while in etcd3 this becomes a ubiquitous model, while maintaining hierarchical compatibility.

Using the containerized version of etcd, we can create the above tree and retrieve it as follows:

$docker run-- rm-d-p 2379VR 2379\

-- name test-etcd3 quay.io/coreos/etcd:v3.1.0 / usr/local/bin/etcd\

-advertise-client-urls http://0.0.0.0:2379-- listen-client-urls http://0.0.0.0:2379

$curl localhost:2379/v2/keys/foo-XPUT-d value= "some value"

$curl localhost:2379/v2/keys/bar/this-XPUT-d value=42

$curl localhost:2379/v2/keys/bar/that-XPUT-d value=take

$http localhost:2379/v2/keys/?recursive=true

HTTP/1.1 200 OK

Content-Length: 327

Content-Type: application/json

Date: Tue, 06 Jun 2017 12:28:28 GMT

X-Etcd-Cluster-Id: 10e5e39849dab251

X-Etcd-Index: 6

X-Raft-Index: 7

X-Raft-Term: 2

{

"action": "get"

"node": {

"dir": true

"nodes": [

{

"createdIndex": 4

"key": "/ foo"

"modifiedIndex": 4

"value": "some value"

}

{

"createdIndex": 5

"dir": true

"key": "/ bar"

"modifiedIndex": 5

"nodes": [

{

"createdIndex": 5

"key": "/ bar/this"

"modifiedIndex": 5

"value": "42"

}

{

"createdIndex": 6

"key": "/ bar/that"

"modifiedIndex": 6

"value": "take"

}

]

}

]

}

}

Now that we have an overview of how etcd works, let's move on to discussing how etcd is used in Kubernetes.

Etcd in the cluster

In Kubernetes, etcd is a slap independent component of the control plane. Before the Kubernetes1.5.2 version, we used the etcd2 version, but after the Kubernetes1.5.2 version, we switched to the etcd3 version. It is worth noting that in the Kubernetes1.5.x version, etcd still uses the v2 API model, after which this will begin to become the v3 API model, including the data model used. From a developer's point of view, this doesn't seem to have a direct impact, because API Server interacts with storage abstractly and doesn't care whether the implementation of back-end storage is etcd v2 or v3. However, from the point of view of the cluster administrator, you still need to know which version of etcd is used, because the cluster administrator needs to backup and restore the data on a daily basis.

You can configure how to use etcd in the relevant startup items of API Server. The parameters of etcd-related startup items of API Server are as follows:

$kube-apiserver-h

...

-- etcd-cafile string SSL Certificate Authority file used to secure etcd communication.

-- etcd-certfile string SSL certification file used to secure etcd communication.

-- etcd-keyfile string SSL keyfile used to secure etcd communication.

...

-- etcd-quorum-read If true, enable quorum read.

Etcd-servers List of etcd servers to connect with (scheme://ip:port)...

...

Kubernetes data stored in etcd is stored in JSON string or Protocol Buffers format. Let's look at a concrete example: create a pod for webserver in the namespace of apiserver-sandbox. Then we use the etcdctl tool to view the relevant etcd (in this link, etcd version 3.1.0) data.

$cat pod.yaml

ApiVersion: v1

Kind: Pod

Metadata:

Name: webserver

Spec:

Containers:

-name: nginx

Image: tomaskral/nonroot-nginx

Ports:

-containerPort: 80

$kubectl create-f pod.yaml

$etcdctl ls /

/ kubernetes.io

/ openshift.io

$etcdctl get / kubernetes.io/pods/apiserver-sandbox/webserver

{

"kind": "Pod"

"apiVersion": "v1"

"metadata": {

"name": "webserver"

...

Let's take a look at how this pod object is finally stored in etcd, via kubectl create-f pod.yaml. The following figure depicts the overall process:

1. The client (such as kubectl) provides an object in an ideal state, such as in YAML format, v1 version.

2. Kubectl converts YAML to JSON format and sends it.

3. API Server performs lossless conversion corresponding to different versions of the same type of object. Fields that do not exist in previous versions are stored in annotations.

4. API Server converts the accepted objects to the specification storage version, which is specified by API Server and is usually the latest stable version, such as v1.

5. Finally, the object is parsed into a value by JSON or protobuf, and stored in etcd through a specific key.

We can determine the format in which we want the serialized data to be stored in etcd by configuring storage-media-type, the startup parameter of kube-apiserver, which is application/vnd.kubernetes.protobuf by default. We can also determine the default version number of each group Group object stored in etcd by configuring the storage-versions startup parameter.

Now let's take a look at how lossless conversion works, which we will enumerate using the Kubernetes object Horizontal Pod Autoscaling (HPA). As the name implies, HPA means to control the scaling of Pod by monitoring the usage of resources combined with ReplicationController.

First we look forward to an API agent (so that we can access it directly locally) and start ReplicationController, as well as HPA.

$kubectl proxy-port=8080 &

$kubectl create-f https://raw.githubusercontent.com/mhausenblas/kbe/master/specs/rcs/rc.yaml

Kubectl autoscale rc rcex-min=2-max=5-cpu-percent=80

Kubectl get hpa/rcex-o yaml

Now, you can use httpie-- and you can also use curl-- to request from API server to get the HPA object using the current stable version (autoscaling/v1) or the previous version (extensions/v1beta1). The difference between the two versions is as follows:

$http localhost:8080/apis/extensions/v1beta1/namespaces/api-server-deepdive/horizontalpodautoscalers/rcex > hpa-v1beta1.json

$http localhost:8080/apis/autoscaling/v1/namespaces/api-server-deepdive/horizontalpodautoscalers/rcex > hpa-v1.json

$diff-u hpa-v1beta1.json hpa-v1.json

{

"kind": "HorizontalPodAutoscaler"

-"apiVersion": "extensions/v1beta1"

+ "apiVersion": "autoscaling/v1"

"metadata": {

"name": "rcex"

"namespace": "api-server-deepdive"

-"selfLink": "/ apis/extensions/v1beta1/namespaces/api-server-deepdive/horizontalpodautoscalers/rcex"

+ "selfLink": "/ apis/autoscaling/v1/namespaces/api-server-deepdive/horizontalpodautoscalers/rcex"

"uid": "ad7efe42-50ed-11e7-9882-5254009543f6"

"resourceVersion": "267762"

"creationTimestamp": "2017-06-14T10:39:00Z"

}

"spec": {

-"scaleRef": {

+ "scaleTargetRef": {

"kind": "ReplicationController"

"name": "rcex"

-"apiVersion": "v1"

-"subresource": "scale"

+ "apiVersion": "v1"

}

"minReplicas": 2

"maxReplicas": 5

-"cpuUtilization": {

-"targetPercentage": 80

-}

+ "targetCPUUtilizationPercentage": 80

We can see that the version of HorizontalPodAutoscale has changed from v1beta1 to v1. API server can be converted losslessly before different versions, no matter which version is actually stored in etcd.

After understanding the entire storage process, let's explore how API server encodes the data and decodes it into etcd in the form of JSON or protobuf, taking into account the version of etcd.

API Server saves all known Kubernetes object types in the Go type registry (registry) named Scheme. In this registry, you define the types of each Kubernetes object and how to convert them, how to create new objects, and how to encode and decode the objects into JSON or protobuf.

When API Server receives an object from the client, such as kubectl, you can know the specific version number of the object through the HTTP path. First, an empty object is created for this object using the corresponding version of Scheme, and then the object content passed by HTTP is decoded and converted through JSON or protobuf. After decoding, an object is created and stored in etcd.

There may be many versions in API, and if you want to support direct conversion between each version, it is often troublesome to deal with. For example, if there are three versions under an API, it should support direct conversion from one version to the other two versions (such as v1 ⇔ v1alpha1, v1 ⇔ v1beta1, v1beta1 ⇔ v1alpha1). To avoid this problem, there is a special "internal" version in API server. When you need to convert between two versions, first convert to the internal version, and then to the corresponding converted version. In this way, each version can be converted indirectly to any other version as long as it supports the ability to convert to the internal version. So an object is first converted to the internal version, then converted to a stable v1 version, and then stored in etcd.

V1beta1 ⇒ internal ⇒ v1

In the first step of the transformation, if some fields are not assigned by the user, these will be assigned as a default value. For example, there is certainly not a new field in the v1 version in v1beta1. In this case, the user certainly cannot assign a value to this field in the v1beta1 version. At this point, in the first step of the transformation, we will assign a default value to this field to generate a valid internal.

Verification and admission

There are two important steps in the conversion process, as shown in the following figure:

V1beta1 ⇒ internal ⇒ | ⇒ | ⇒ v1 ⇒ json/yaml ⇒ etcd

Admission validation

Admission and validation are steps that must be passed before creating and updating objects are stored in etcd. Some of their rules are as follows:

1. Admission: see if some constraints in the cluster allow this object to be created or updated, and set some default values for the object based on the relevant configuration of the cluster. There are many such constraints in Kubernetes. Here are some examples:

NamespaceLifecycle: if the namespace does not exist, all incoming requests under that namespace are rejected.

LimitRanger: forces restrictions on the usage of resources in the namespace.

ServiceAccount: create a service account for pod.

DefaultStorageClass: if the user does not assign a value to PersistentVolumeClaims, set it to a default value.

ResourceQuota: enforce quota constraints on current users on the cluster, and requests may be denied if the quota is insufficient.

two。 Validation: checks whether the incoming object (during creation and update) is in a valid format and whether the related values are valid. For example:

1. Check to see if the required fields are filled.

two。 Check that the string is in the correct format (for example, only lowercase is allowed).

3. Whether there are conflicts in some fields (for example, there are two containers with the same names).

Validation does not care about other types of object instances, in other words, it only cares about the static checking of each object, regardless of the cluster configuration.

Admission can be enabled or disabled with flag-- admission-control=. Most of them can have cluster management configurations. In addition, in Kubernetes 1.7, the webhook mechanism can be used to extend the admission mechanism, and the controller can be used to implement the traditional validation of objects.

Migrate storage object

Final note on storage object migration: when Kubernetes needs to upgrade to a new version, it is critical to back up the relevant cluster data according to the relevant documentation steps for each version. On the one hand, this is due to the transformation from etcd2 to etcd3, and on the other hand, it is due to the continuous development of Kind and version of Kubernetes objects.

In etcd, each object exists in the preferred storage version (preferred storage version). However, over time, objects in etcd storage may exist in a very old version. If the object version is obsolete at some point in the future, its protobuf or JSON will no longer be decoded. Therefore, this data needs to be rewritten and migrated before the cluster upgrade.

Thank you for your reading. I believe you have some understanding of "how to manage the state of Kubernetes objects". Go ahead and practice it. If you want to know more about it, you can follow the website! The editor will continue to bring you better articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.