Shulou(Shulou.com)05/31 Report--

Today, I will talk to you about the principle of Kubernetes APIServer, which may not be well understood by many people. in order to make you understand better, the editor has summarized the following content for you. I hope you can get something according to this article.

Preface

The whole Kubernetes technology system is composed of declarative API and Controller, and kube-apiserver is the declarative apiserver of Kubernetes, and provides a bridge for the interaction of other components. Therefore, it is very important to deepen the understanding of kube-apiserver.

Overall component function

Kube-apiserver, as the only entry for the etcd operation of the entire Kubernetes cluster, is responsible for the authentication, verification and CRUD operations of Kubernetes resources, and provides RESTful APIs for other components to call:

Kube-apiserver contains three types of APIServer:

AggregatorServer: handles APIService resource requests under the apiregistration.k8s.io group, while intercepting and forwarding requests from users to aggregated server (AA)

KubeAPIServer: responsible for some general processing of requests, including authentication, authentication and REST services for built-in resources (pod, deployment,service and etc), etc.

ApiExtensionsServer: responsible for registering CustomResourceDefinition (CRD) apiResources and apiVersions, while processing REST requests for CRD and corresponding CustomResource (CR) (404 will be returned if the corresponding CR cannot be processed), which is also the last link of apiserver Delegation

It also includes bootstrap-controller, which is mainly responsible for the creation and management of Kubernetes default apiserver service.

Next, we will give an overview of the above components.

Bootstrap-controller

Apiserver bootstrap-controller creation & runs the logic in the k8s.io/kubernetes/pkg/master directory

Bootstrap-controller is mainly used to create and maintain internal kubernetes default apiserver service

Kubernetes default apiserver service spec.selector is empty, which is the biggest difference between default apiserver service and other normal service, indicating that the endpoints corresponding to this special service is not controlled by endpoints controller, but is directly managed by kube-apiserver bootstrap-controller (maintained by this code, not by the pod selector).

Several main functions of bootstrap-controller are as follows:

Create default, kube-system, and kube-public and kube-node-lease namespaces

Create & maintain kubernetes default apiserver service and corresponding endpoint

Provide Service ClusterIP-based inspection and repair functions (--service-cluster-ip-range specified range)

Provide Service NodePort-based inspection and repair functions (--service-node-port-range specified range)

/ / k8s.io/kubernetes/pkg/master/controller.go:142// Start begins the core controller loops that must exist for bootstrapping// a cluster.func (c * Controller) Start () {if c.runner! = nil {return} / / Reconcile during first run removing itself until server is ready. EndpointPorts: = createEndpointPortSpec (c.PublicServicePort, "https", c.ExtraEndpointPorts) if err: = c.EndpointReconciler.RemoveEndpoints (kubernetesServiceName, c.PublicIP, endpointPorts) Err! = nil {klog.Errorf ("Unable to remove old endpoints from kubernetes service:% v", err)} repairClusterIPs: = servicecontroller.NewRepair (c.ServiceClusterIPInterval, c.ServiceClient, c.EventClient, & c.ServiceClusterIPRange, c.ServiceClusterIPRegistry, & c.SecondaryServiceClusterIPRange, c.SecondaryServiceClusterIPRegistry) repairNodePorts: = portallocatorcontroller.NewRepair (c.ServiceNodePortInterval, c.ServiceClient, c.EventClient, c.ServiceNodePortRange, c.ServiceNodePortRegistry) / / run all of the controllers once prior to returning from Start. If err: = repairClusterIPs.RunOnce (); err! = nil {/ / If we fail to repair cluster IPs apiserver is useless. We should restart and retry. Klog.Fatalf ("Unable to perform initial IP allocation check:% v", err)} if err: = repairNodePorts.RunOnce (); err! = nil {/ / If we fail to repair node ports apiserver is useless. We should restart and retry. Klog.Fatalf ("Unable to perform initial service nodePort check:% v", err)} / / perform the four main functions of bootstrap controller regularly (reconciliation) c.runner = async.NewRunner (c.RunKubernetesNamespaces, c.RunKubernetesService, repairClusterIPs.RunUntil, repairNodePorts.RunUntil) c.runner.Start ()}

For more details on how the code works, refer to kubernetes-reading-notes.

KubeAPIServer

KubeAPIServer mainly provides operation requests for built-in API Resources, registers routing information for each API Resources in Kubernetes, and exposes RESTful API, so that services in and outside the cluster can operate resources in Kubernetes through RESTful API.

In addition, kubeAPIServer is the core of the whole Kubernetes apiserver. The aggregatorServer and apiExtensionsServer described below are extended on the basis of kubeAPIServer (supplemented by Kubernetes's ability to customize resources).

The core function of kubeAPIServer is to add routes to Kubernetes built-in resources, as follows:

Call m.InstallLegacyAPI to add the core API Resources to the route, which is the resource that starts with / api in apiserver

Call m.InstallAPIs to add the extended API Resources to the route, which is the resource that starts with / apis in apiserver

/ / k8s.io/kubernetes/pkg/master/master.go:332// New returns a new instance of Master from the given config.// Certain config fields will be set to a default value if unset.// Certain config fields must be specified, including:// KubeletClientConfigfunc (c completedConfig) New (delegationTarget genericapiserver.DelegationTarget) (* Master Error) {. / / install LegacyAPI (core API) / / install legacy rest storage if c.ExtraConfig.APIResourceConfigSource.VersionEnabled (apiv1.SchemeGroupVersion) {legacyRESTStorageProvider: = corerest.LegacyRESTStorageProvider {StorageFactory: c.ExtraConfig.StorageFactory, ProxyTransport: c.ExtraConfig.ProxyTransport, KubeletClientConfig: c.ExtraConfig.KubeletClientConfig EventTTL: c.ExtraConfig.EventTTL, ServiceIPRange: c.ExtraConfig.ServiceIPRange, SecondaryServiceIPRange: c.ExtraConfig.SecondaryServiceIPRange, ServiceNodePortRange: c.ExtraConfig.ServiceNodePortRange, LoopbackClientConfig: c.GenericConfig.LoopbackClientConfig, ServiceAccountIssuer: c.ExtraConfig.ServiceAccountIssuer, ServiceAccountMaxExpiration: c.ExtraConfig.ServiceAccountMaxExpiration APIAudiences: c.GenericConfig.Authentication.APIAudiences,} if err: m.InstallLegacyAPI (& c, c.GenericConfig.RESTOptionsGetter, legacyRESTStorageProvider) Err! = nil {return nil, err}}... / install APIs (named groups apis) if err: = m.InstallAPIs (c.ExtraConfig.APIResourceConfigSource, c.GenericConfig.RESTOptionsGetter, restStorageProviders...); err! = nil {return nil, err}. Return m, nil}

The entire kubeAPIServer provides three types of API Resource interfaces:

Core group: mainly under / api/v1

Named groups: its path is / apis/$GROUP/$VERSION

Some API of system status, such as / metrics, / version, etc.

The URL of API is roughly composed of / apis/ {group} / {version} / namespaces/ {namespace} / resource/ {name}, and the structure is shown in the following figure:

The purpose of kubeAPIServer creating a corresponding RESTStorage,RESTStorage for each API resource is to correspond the access path of each resource and the operations stored at its backend: determine which operations the resource can perform (such as create, update, etc.) through the API implemented by the constructed RESTStorage, and store the corresponding operations in action. Each operation corresponds to a standard REST method, such as create corresponds to POST, and update corresponds to REST method PUT. Finally, iterate through the actions array in turn, add a handler (handler corresponds to the relevant APIs implemented by REST Storage) for each operation, register it to route, and finally provide RESTful API as follows:

/ / m.GenericAPIServer.InstallLegacyAPIGroup-- > s.installAPIResources-- > apiGroupVersion.InstallREST-- > installer.Install-- > a.registerResourceHandlers// k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/endpoints/installer.go:181func (a * APIInstaller) registerResourceHandlers (path string, storage rest.Storage, ws * restful.WebService) (* metav1.APIResource, error) {. / / 1, determine which REST operation interfaces are implemented by the resource To determine which verbs it supports in order to add routes / / what verbs are supported by the storage, used to know what verbs we support per path creater, isCreater: = storage. (rest.Creater) namedCreater, isNamedCreater: = storage. (rest.NamedCreater) lister, isLister: = storage. (rest.Lister) getter IsGetter: = storage. (rest.Getter). / / 2, add the corresponding actions for resource (+ depending on whether namespace is supported) / / Get the list of actions for the given scope. Switch {case! namespaceScoped: / / Handle non-namespace scoped resources like nodes. ResourcePath: = resource resourceParams: = params itemPath: = resourcePath + "/ {name}" nameParams: = append (params, nameParam) proxyParams: = append (nameParams, pathParam)... / / Handler for standard REST verbs (GET, PUT, POST and DELETE). / / Add actions at the resource path: / api/apiVersion/resource actions = appendIf (actions, action {"LIST", resourcePath, resourceParams, namer, false}, isLister) actions = appendIf (actions, action {"POST", resourcePath, resourceParams, namer, false}, isCreater). / / 3, mapping from rest.Storage to restful.Route / / adding the corresponding handler for _ for each operation Action: = range actions {... Switch action.Verb {... Case "POST": / / Create a resource. Var handler restful.RouteFunction / / 4. Initialize handler if isNamedCreater {handler = restfulCreateNamedResource (namedCreater, reqScope, admit)} else {handler = restfulCreateResource (creater, reqScope, admit)} handler = metrics.InstrumentRouteFunc (action.Verb, group, version, resource, subresource, requestScope, metrics.APIServerComponent Handler). / / 5. Bind route to handler route: = ws.POST (action.Path) .To (handler). Doc (doc). Param (ws.QueryParameter ("pretty", "If 'true', then the output is pretty printed."). Operation ("create" + namespaced+kind+strings.Title (subresource) + operationSuffix). Produces (append (storageMeta.ProducesMIMETypes (action.Verb), mediaTypes...). Returns (http.StatusOK, "OK", producedObject). / / TODO: in some cases, the API may return a v1.Status instead of the versioned object / / but currently go-restful can't handle multiple different objects being returned. Returns (http.StatusCreated, "Created", producedObject). Returns (http.StatusAccepted, "Accepted", producedObject). Reads (defaultVersionedObject). Writes (producedObject) if err: = AddObjectParams (ws, route, versionedCreateOptions); err! = nil {return nil, err} addParams (route, action.Params) / / 6, add to the route routes = append (routes, route) case "DELETE": / / Delete a resource. ... Default: return nil, fmt.Errorf ("unrecognized action verb:% s", action.Verb)} for _, route: = range routes {route.Metadata (ROUTE_META_GVK, metav1.GroupVersionKind {Group: reqScope.Kind.Group, Version: reqScope.Kind.Version, Kind: reqScope.Kind.Kind ) route.Metadata (ROUTE_META_ACTION, strings.ToLower (action.Verb)) ws.Route (route)} / / Note: update GetAuthorizerAttributes () when adding a custom handler. }...}

The kubeAPIServer code is organized as follows:

1. Apiserver whole startup logic k8s.io/kubernetes/cmd/kube-apiserver2. Apiserver bootstrap-controller create & run logic k8s.io/kubernetes/pkg/master3. API Resource corresponds to the backend RESTStorage (based on genericregistry.Store) to create k8s.io/kubernetes/pkg/registry4. Aggregated-apiserver creation & processing logic k8s.io/kubernetes/staging/src/k8s.io/kube-aggregator5. Extensions-apiserver creation & processing logic k8s.io/kubernetes/staging/src/k8s.io/apiextensions-apiserver6. Create apiserver & run k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/server7. Register the API Resource resource processing handler (InstallREST&Install ®isterResourceHandlers) k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/endpoints8. Create storage backend (etcdv3) k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/storage9. Genericregistry.Store.CompleteWithOptions initializes k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/registry

The call chain is sorted out as follows:

For more details on how the code works, refer to kubernetes-reading-notes.

AggregatorServer

AggregatorServer is mainly used to handle Aggregated APIServer (AA), the second way to extend Kubernetes API Resources, to proxy CR requests to AA:

Here, combined with the official aggregated apiserver example sample-apiserver given by Kubernetes, the principle is summarized as follows:

AggregatorServer forwards the request by associating the APIServices object to a Service, and the associated Service type further determines the form of request forwarding. AggregatorServer includes a GenericAPIServer and a Controller that maintains its own state. GenericAPIServer mainly handles APIService resource requests under the apiregistration.k8s.io group, while Controller includes:

ApiserviceRegistrationController: responsible for building an agent based on the aggregated server service defined by APIService and forwarding the CR request to the backend aggregated server

AvailableConditionController: maintains the availability of APIServices, including whether its reference Service is available, etc.

AutoRegistrationController: a specific set of APIServices used to maintain the presence of the API

CrdRegistrationController: responsible for automatically registering CRD GroupVersions with APIServices

OpenAPIAggregationController: synchronizes changes in APIServices resources to the provided OpenAPI document

ApiserviceRegistrationController is responsible for building the proxy based on the aggregated server service defined by APIService and forwarding the request from CR to the back-end aggregated server. There are two types of apiService: Local (Service is empty) and Service (Service is not empty). ApiserviceRegistrationController is responsible for setting proxies for these two types of apiService: the Local type is routed directly to kube-apiserver for processing. The Service type sets the proxy and converts the request into a request for aggregated Service (proxyPath: = "/ apis/" + apiService.Spec.Group + "/" + apiService.Spec.Version), while the requested cloud load balancer policy is to give priority to local access to kube-apiserver (if service is kubernetes default apiserver service:443) = > access through service ClusterIP:Port (default) or by randomly selecting service endpoint backend:

Func (s * APIAggregator) AddAPIService (apiService * v1.APIService) error {... ProxyPath: = "/ apis/" + apiService.Spec.Group + "/" + apiService.Spec.Version / / v1. Is a special case for the legacy API. It proxies to a wider set of endpoints. If apiService.Name = = legacyAPIServiceName {proxyPath = "/ api"} / / register the proxy handler proxyHandler: = & proxyHandler {localDelegate: s.delegateHandler, proxyClientCert: s.proxyClientCert, proxyClientKey: s.proxyClientKey, proxyTransport: s.proxyTransport, serviceResolver: s.serviceResolver, egressSelector: s.egressSelector,}. S.proxyHandlers [apiService.Name] = proxyHandler s.GenericAPIServer.Handler.NonGoRestfulMux.Handle (proxyPath, proxyHandler) s.GenericAPIServer.Handler.NonGoRestfulMux.UnlistedHandlePrefix (proxyPath+ "/", proxyHandler). / / it's time to register the group aggregation endpoint groupPath: = "/ apis/" + apiService.Spec.Group groupDiscoveryHandler: = & apiGroupHandler {codecs: aggregatorscheme.Codecs, groupName: apiService.Spec.Group, lister: s.lister, delegate: s.delegateHandler } / / aggregation is protected s.GenericAPIServer.Handler.NonGoRestfulMux.Handle (groupPath, groupDiscoveryHandler) s.GenericAPIServer.Handler.NonGoRestfulMux.UnlistedHandle (groupPath+ "/", groupDiscoveryHandler) s.handledGroups.Insert (apiService.Spec.Group) return nil} / / k8s.io/kubernetes/staging/src/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:109func (r * proxyHandler) ServeHTTP (w http.ResponseWriter Req * http.Request) {/ / load roxyHandlingInfo processing request value: = r.handlingInfo.Load () if value = = nil {r.localDelegate.ServeHTTP (w, req) return} handlingInfo: = value. (proxyHandlingInfo). / / determine whether the APIService service is normal if! handlingInfo.serviceAvailable {proxyError (w, req, "service unavailable" Http.StatusServiceUnavailable) return} / / convert the original request into a request for APIService / / write a new location based on the existing request pointed at the target service location: = & url.URL {} location.Scheme = "https" rloc, err: = r.serviceResolver.ResolveEndpoint (handlingInfo.serviceNamespace, handlingInfo.serviceName, handlingInfo.servicePort) if err! = nil {klog.Errorf ("error resolving% s:% v", handlingInfo.serviceNamespace, handlingInfo.serviceName Err) proxyError (w, req, "service unavailable", http.StatusServiceUnavailable) return} location.Host = rloc.Host location.Path = req.URL.Path location.RawQuery = req.URL.Query () .Encode () newReq, cancelFn: = newRequestForProxy (location, req) defer cancelFn (). ProxyRoundTripper = transport.NewAuthProxyRoundTripper (user.GetName (), user.GetGroups (), user.GetExtra (), proxyRoundTripper) handler: = proxy.NewUpgradeAwareHandler (location, proxyRoundTripper, true, upgrade, & responder {w: W}) handler.ServeHTTP (w, newReq)}

$kubectl get APIService NAME SERVICE AVAILABLE AGE...v1.apps Local True 50d...v1beta1.metrics.k8s.io kube-system/metrics-server True 50d...

# default APIServices$ kubectl get-o yaml APIService/v1.appsapiVersion: apiregistration.k8s.io/v1kind: APIServicemetadata: labels: kube-aggregator.kubernetes.io/automanaged: onstart name: v1.apps selfLink: / apis/apiregistration.k8s.io/v1/apiservices/v1.appsspec: apps groupPriorityMinimum: 17800 version: V1 versionPriority: 15status: conditions:-lastTransitionTime: "2020-10-20T10:39:48Z" message: Local APIServices are always available reason: Local status: "True" Type: Available# aggregated server $kubectl get-o yaml APIService/v1beta1.metrics.k8s.ioapiVersion: apiregistration.k8s.io/v1kind: APIServicemetadata: labels: addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/cluster-service: "true" name: v1beta1.metrics.k8s.io selfLink: / apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.iospec: group: metrics.k8s.io groupPriorityMinimum: 100 insecureSkipTLSVerify: true service: name: metrics-server namespace: Kube-system port: 443 version: v1beta1 versionPriority: 100status: conditions:-lastTransitionTime: "2020-12-05T00:50:48Z" message: all checks passed reason: Passed status: "True" type: Available# CRD$ kubectl get-o yaml APIService/v1.duyanghao.example.comapiVersion: apiregistration.k8s.io/v1kind: APIServicemetadata: labels: kube-aggregator.kubernetes.io/automanaged: "true" name: v1.duyanghao.example.com selfLink: / apis/apiregistration K8s.io/v1/apiservices/v1.duyanghao.example.comspec: group: duyanghao.example.com groupPriorityMinimum: 1000 version: v1 versionPriority: 100status: conditions:-lastTransitionTime: "2020-12-11T08:45:37Z" message: Local APIServices are always available reason: Local status: "True" type: Available

During aggregatorServer creation, a default APIService list is created based on all the API resources defined by kube-apiserver, named $VERSION.$GROUP, and these APIService will be labeled kube-aggregator.kubernetes.io/automanaged: onstart, for example: v1.apps apiService. AutoRegistrationController creates and maintains the APIService in these lists, that is, the Local apiService; we see, does not deal with custom APIService (aggregated server).

Aggregated server implements the CRUD API interface of CR (custom API resources), and can flexibly choose backend storage, share etcd with core kube-apiserver, or deploy etcd database or other databases independently. The CR API path of aggregated server implementation is: / apis/$GROUP/$VERSION, and the specific sample apiserver is: / apis/wardle.example.com/v1alpha1. The following resource types are: flunders and fischers

Aggregated server implements integration and interaction with core kube-apiserver by deploying APIService-type resources, and service fields points to the corresponding aggregated server service.

The structure of the sample-apiserver directory is as follows. You can refer to writing your own aggregated server:

Staging/src/k8s.io/sample-apiserver ├── artifacts │ ├── example │ │ ├── apiservice.yaml... ├── hack ├── main.go └── pkg ├── admission ├── apis ├── apiserver ├── cmd ├── generated │ ├── clientset │ │ └── versioned... typed └── wardle │ │ ├── v1alpha1 │ │ └── v1beta1 │ ├── informers │ │ └── externalversions │ │ └── wardle │ │ ├── v1alpha1 │ v1beta1 listers wardle ├── v1alpha1 │ │ └── v1beta1 └── registry

Where artifacts is used to deploy the yaml example

Hack directory stores automatic scripts (eg: update-codegen)

Main.go is the aggregated server startup portal; pkg/cmd is responsible for launching aggregated server specific logic; pkg/apiserver is used for aggregated server initialization and route registration

Pkg/apis is responsible for defining the structure of related CR and automatically generating (update-codegen).

Pkg/admission is responsible for admittance related code

Pkg/generated is responsible for generating clientset,informers that accesses CR, as well as listers

The pkg/registry directory is responsible for CR-related RESTStorage implementation

For more details on how the code works, refer to kubernetes-reading-notes.

ApiExtensionsServer

ApiExtensionsServer is mainly responsible for the registration of CustomResourceDefinition (CRD) apiResources and apiVersions, while processing REST requests for CRD and corresponding CustomResource (CR) (404 will be returned if the corresponding CR cannot be processed), which is also the last link of apiserver Delegation.

The principles are summarized as follows:

Custom Resource, referred to as CR, is a custom resource type of Kubernetes, corresponding to various resource types built into Kubernetes, such as Pod, Service and so on. With CR, we can define any resource type we want.

CRD registers CR to Kubernetes in the form of yaml file to implement custom api-resources, which belongs to the second way to expand Kubernetes API resources and is also a widely used one.

APIExtensionServer is responsible for the registration of CustomResourceDefinition (CRD) apiResources and apiVersions, and processes the REST requests of CRD and corresponding CustomResource (CR) (404 will be returned if the corresponding CR cannot be processed), which is also the last link of apiserver Delegation.

CrdRegistrationController is responsible for automatically registering CRD GroupVersions with APIServices. The specific logic is: enumerate all the CRDs, then build the APIService based on the crd.Spec.Group and crd.Spec.Versions fields defined by CRD, and add it to the autoRegisterController.apiServicesToSync, which is created and maintained by autoRegisterController. This is why the corresponding APIService object is generated after the CRD is created.

The controller and features included in APIExtensionServer are as follows:

-`namingController`: check whether there are naming conflicts in crd obj, which can be checked in crd `.status.conditions`;-`establishingController`: check whether crd is normal, which can be checked in crd `.status.conditions`;-`nonStructuralSchemaController`: check whether crd obj structure is normal, which can be checked in crd `.status.conditions`;-`apiApprovalController`: check whether crd follows the Kubernetes API declaration policy, which can be checked in crd `.status.conditions` -`finalizingController`: similar to finalizes, related to the deletion of CRs

OpenapiController: synchronize changes in crd resources to the provided OpenAPI document, which can be viewed by visiting / openapi/v2

CrdController: responsible for registering crd information with apiVersions and apiResources, which can be viewed through kubectl api-versions and kubectl api-resources

The kubectl api-versions command returns version information for all Kubernetes cluster resources (actually issued two requests, https://127.0.0.1:6443/api and https://127.0.0.1:6443/apis, and finally merges the returned results of the two requests)

$kubectl-vault 8 api-versions I1211 11 loader.go:375 44 loader.go:375] Config loaded from file: / root/.kube/configI1211 11 44 loader.go:375 50.277005 22493 round_trippers.go:420] GET https://127.0.0.1:6443/api?timeout=32s...I1211 11 44 loader.go:375 50.290265 22493 request.go:1068] Response Body: {"kind": "APIVersions", "versions": ["v1"] "serverAddressByClientCIDRs": [{"clientCIDR": "0.0.0.0 x.x.x.x:6443" 0 "," serverAddress ":" x.x.x.x:6443 "}]} I1211 11 44 apiVersion 50.293673 22493 round_trippers.go:420] GET https://127.0.0.1:6443/apis?timeout=32s...I1211 11 44V 50.298360 22493 request.go:1068] Response Body: {" kind ":" APIGroupList "," apiVersion ":" v1 " "groups": [{"name": "apiregistration.k8s.io", "versions": [{"groupVersion": "apiregistration.k8s.io/v1", "version": "v1"}, {"groupVersion": "apiregistration.k8s.io/v1beta1", "version": "v1beta1"}], "preferredVersion": {"groupVersion": "apiregistration.k8s.io/v1", "version": "v1"}}, {"name": "extensions", "versions": [{"groupVersion": "extensions/v1beta1" "version": "v1beta1"}], "preferredVersion": {"groupVersion": "extensions/v1beta1", "version": "v1beta1"}, {"name": "apps", "versions": [{"groupVersion": "apps/v1", "version": "v1"}], "preferredVersion": {"groupVersion": "apps/v1", "version": "v1"}, {"name": "events.k8s.io", "versions": [{"groupVersion": "events.k8s.io/v1beta1" "version": "v1beta1"}], "preferredVersion": {"groupVersion": "events.k8s.io/v1beta1", "version": "v1beta1"}, {"name": "authentication.k8s.io", "versions": [{"groupVersion": "authentication.k8s.io/v1", "version": "v1"}, {"groupVersion": "authentication.k8s.io/v1beta1", "version": "v1beta1"}], "preferredVersion": {"groupVersion": "authentication.k8s.io/v1" "[truncated 4985 chars] apiextensions.k8s.io/v1apiextensions.k8s.io/v1beta1apiregistration.k8s.io/v1apiregistration.k8s.io/v1beta1apps/v1authentication.k8s.io/v1beta1...storage.k8s.io/v1storage.k8s.io/v1beta1v1

The kubectl api-resources command first gets all the API version information, and then calls the API for each API version to get all API resource types under that version.

$kubectl-vault 8 api-resources 5077 loader.go:375] Config loaded from file: / root/.kube/config I1211 15 loader.go:375 19V 47.593450 15077 round_trippers.go:420] GET https://127.0.0.1:6443/api?timeout=32s I1211 15 V 19V 47.602273 15077 request.go:1068] Response Body: {"kind": "APIVersions", "versions": ["v1"], "serverAddressByClientCIDRs": [{"clientCIDR": "0.0.0.0Univer" "serverAddress": "x.x.x.x:6443"}]} I1211 15round_trippers.go:420 19round_trippers.go:420 47.606279 15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis?timeout=32s I1211 15round_trippers.go:420 19round_trippers.go:420 47.610333 15077 request.go:1068] Response Body: {"kind": "APIGroupList", "apiVersion": "v1", "groups": [{"name": "apiregistration.k8s.io", "versions": [{"groupVersion": "apiregistration.k8s.io/v1" "version": "v1"}, {"groupVersion": "apiregistration.k8s.io/v1beta1", "version": "v1beta1"}], "preferredVersion": {"groupVersion": "apiregistration.k8s.io/v1", "version": "v1"}}, {"name": "extensions", "versions": [{"groupVersion": "extensions/v1beta1", "version": "v1beta1"}], "preferredVersion": {"groupVersion": "extensions/v1beta1", "version": "v1beta1"}}, {"name": "apps"} "versions": [{"groupVersion": "apps/v1", "version": "v1"}], "preferredVersion": {"groupVersion": "apps/v1", "version": "v1"}}, {"name": "events.k8s.io", "versions": [{"groupVersion": "events.k8s.io/v1beta1", "version": "v1beta1"}], "preferredVersion": {"groupVersion": "events.k8s.io/v1beta1", "version": "v1beta1"} {"name": "authentication.k8s.io", "versions": [{"groupVersion": "authentication.k8s.io/v1", "version": "v1"}, {"groupVersion": "authentication.k8s.io/v1beta1", "version": "v1beta1"}], "preferredVersion": {"groupVersion": "authentication.k8s.io/v1" "[truncated 4985 chars] I1211 15 https://127.0.0. 19 truncated 47.614700 15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/batch/v1?timeout=32s I1211 15 https://127.0.0. 19V 47.614804 15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/authentication.k8s.io/v1?timeout=32s I1211 15 15 round_trippers.go:420 47.615687 15077 GET 1:6443/apis/auth.tkestack.io/v1?timeout=32s https://127.0.0.1:6443/apis/authentication.k8s.io/v1beta1?timeout=32s I1211 15:19:47.616794 15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/coordination.k8s.io/v1?timeout=32s I1211 15:19:47.616863 15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/apps/v1?timeout=32s... NAME SHORTNAMES APIGROUP NAMESPACED KIND bindings true Binding endpoints ep true Endpoints events Ev true Event limitranges limits true LimitRange namespaces ns false Namespace nodes No false Node...

The CR CRUD APIServer processing logic is summarized as follows:

Parse req (GET / apis/duyanghao.example.com/v1/namespaces/default/students) and get the corresponding CRD content (crd, err: = r.crdLister.Get (crdName)) according to group (duyanghao.example.com), version (v1), and resource field (students) in the request path

Obtain crdInfo through crd.UID and crd.Name, and create a corresponding crdInfo (crdInfo, err: = r.getOrCreateServingInfoFor (crd.UID, crd.Name)) if it does not exist. The crdInfo contains the CRD definition and the customresource.REST storage of the CRD corresponding to the Custom Resource

Customresource.REST storage is created by Group (duyanghao.example.com), Version (v1), Kind (Student), Resource (students) corresponding to CR. Since there is no specific structure definition of CR in the Kubernetes code, a generic structure Unstructured (used to save all types of Custom Resource) will be initialized first, and SetGroupVersionKind operation will be performed on the structure (set specific Custom Resource Type).

After getting the Unstructured structure from customresource.REST storage, it will be converted accordingly and returned

/ / k8s.io/kubernetes/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go:223func (r * crdHandler) ServeHTTP (w http.ResponseWriter, req * http.Request) {ctx: = req.Context () requestInfo, ok: = apirequest.RequestInfoFrom (ctx). CrdName: = requestInfo.Resource + "." + requestInfo.APIGroup crd, err: = r.crdLister.Get (crdName). CrdInfo, err: = r.getOrCreateServingInfoFor (crd.UID, crd.Name) verb: = strings.ToUpper (requestInfo.Verb) resource: = requestInfo.Resource subresource: = requestInfo.Subresource scope: = metrics.CleanScope (requestInfo). Switch {case subresource = = "status" & & subresources! = nil & & subresources.Status! = nil: handlerFunc = r.serveStatus (w, req, requestInfo, crdInfo, terminating, supportedTypes) case subresource = = "scale" & & subresources! = nil & & subresources.Scale! = nil: handlerFunc = r.serveScale (w, req, requestInfo, crdInfo, terminating, supportedTypes) case len (subresource) = 0: handlerFunc = r.serveResource (w, req, requestInfo, crdInfo, terminating) SupportedTypes) default: responsewriters.ErrorNegotiated (schema.GroupResource {Group: requestInfo.APIGroup, Resource: requestInfo.Resource}, requestInfo.Name), Codecs, schema.GroupVersion {Group: requestInfo.APIGroup, Version: requestInfo.APIVersion}, w, req,)} if handlerFunc! = nil {handlerFunc = metrics.InstrumentHandlerFunc (verb, requestInfo.APIGroup, requestInfo.APIVersion, resource, subresource, scope, metrics.APIServerComponent, handlerFunc) handler: = genericfilters.WithWaitGroup (handlerFunc, longRunningFilter CrdInfo.waitGroup) handler.ServeHTTP (w, req) return}}

CreateAPIExtensionsServer= > NewCustomResourceDefinitionHandler= > crdHandler= > Register CR CRUD API API:

/ / New returns a new instance of CustomResourceDefinitions from the given config.func (c completedConfig) New (delegationTarget genericapiserver.DelegationTarget) (* CustomResourceDefinitions, error) {. CrdHandler, err: = NewCustomResourceDefinitionHandler (versionDiscoveryHandler, groupDiscoveryHandler, s.Informers.Apiextensions (). V1 (). CustomResourceDefinitions (), delegateHandler, c.ExtraConfig.CRDRESTOptionsGetter, c.GenericConfig.AdmissionControl, establishingController, c.ExtraConfig.ServiceResolver, c.ExtraConfig.AuthResolverWrapper, c.ExtraConfig.MasterCount, s.GenericAPIServer.Authorizer, c.GenericConfig.RequestTimeout Time.Duration (c.GenericConfig.MinRequestTimeout) * time.Second, apiGroupInfo.StaticOpenAPISpec, c.GenericConfig.MaxRequestBodyBytes,) if err! = nil {return nil, err} s.GenericAPIServer.Handler.NonGoRestfulMux.Handle ("/ apis", crdHandler) s.GenericAPIServer.Handler.NonGoRestfulMux.HandlePrefix ("/ apis/", crdHandler). Return s, nil} Conclusion

This paper gives an overview of Kubernetes apiserver from the source code level, including: aggregatorServer,kubeAPIServer,apiExtensionsServer and bootstrap-controller.

After reading the above, do you have any further understanding of the principle of Kubernetes APIServer? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.