Shulou(Shulou.com)05/31 Report--

This article mainly introduces the relevant knowledge of how to achieve lite-apiserver, the content is detailed and easy to understand, the operation is simple and fast, and has a certain reference value. I believe you will gain something after reading this article on how to achieve lite-apiserver. Let's take a look at it.

Introduction

In version 0.2.0 of SuperEdge, lite-apiserver has undergone major architectural upgrades and feature enhancements. This paper will analyze the edge autonomy of lite-apiserver from the point of view of the implementation of SuperEdge and its cooperation with other SuperEdge components, so as to provide reference for everyone's research and selection.

Edge node autonomy

In the cloud-edge collaborative edge computing scene, the edge nodes are connected to the cloud through the public network. There are many edge nodes, the network environment is complex, and the network quality is uneven. In the case of weak network or disconnection with the cloud, the edge node continues to work normally, and the running business is not affected, so as to achieve the goal of edge node autonomy. In order to achieve edge node autonomy, you need to deal with the following scenarios:

The edge node is disconnected from the cloud, but it is normal. The business container running above should not be expelled, and no new business container should be dispatched to this node.

When the edge node is disconnected from the cloud, the Kubernetes components and business containers on the edge node can continue to run

When the edge node is disconnected from the cloud, after the edge node is restarted, the Kubernetes components and business containers on the node can run

After the edge node and the cloud are restored, the data on the edge node is consistent with that of the cloud.

SuperEdge uses the distributed node health check component edge-health to handle scenario 1 and lite-apiserver to deal with scenarios 2, 3, and 4.

Lite-apiserver is a lightweight apiserver running on the edge node, which proxies all the components and business containers on the node to access the cloud kube-apiserver request and caches the request result efficiently. In the case of cloud edge disconnection, use these caches to provide services to achieve edge autonomy.

Lite-apiserver design features

In addition to meeting the functional requirements of edge node autonomy, lite-apiserver also needs to meet the following design features:

All Client types are supported

As the only "exit" to access cloud kube-apiserver on edge nodes, lite-apiserver needs to support all types of Client, including Kubernetes components running in the form of bin (such as kubelet, etc.) or pod (such as flannel\ kube-proxy, etc.), as well as business containers that InCluster access to kube-apiserver. Further, if the network environment of the edge node is special, and you need to use proxies and other methods to access the cloud kube-apiserver, you only need to set a proxy for the lite-apiserver, and all components can access the cloud kube-apiserver normally, and each component does not need to be configured separately.

Support for caching all types of resources

Support for caching all types of resources, Kubernetes built-in resources and Custom Resources. The Kubernetes component running on the edge node and the request kube-apiserver of the business container have a variety of resources. If only some resource types are cached or only Kubernetes built-in resource types are supported, when the cloud edge is disconnected, the component or business may fail because the corresponding cache cannot be read, thus failing to achieve the autonomous effect of the edge node. Of course, caching that supports all types of resources (especially Custom Resources) also poses challenges for data parsing and processing.

Safety

Edge nodes are widely distributed and the environment is complex, which is more likely to cause security risks. Security issues are also getting more and more attention in edge computing and Kubernetes management. It is a common access control scheme to give lite-apiserver an access right. All requests of its proxy throw away their own permissions and use the permissions of lite-apiserver to access kube-apiserver in the cloud. Since lite-apiserver needs to access and process all types of resources, this permission must be a "super" permission. In this case, malicious programs on one of the edge nodes can operate on all the resources of the cluster through lite-apiserver, which may maliciously destroy the whole cluster. Therefore, from a security point of view, lite-apiserver should not have a "super" permission by design. It can use the original authentication and authentication methods of Kubernetes components and business containers to access cloud kube-apiserver.

Support for multiple cache storage

According to IDC's definition of edge computing layering, the edge is divided into Heavy Edge (edge data center) and Light Edge (low power computing platform). For different scenarios, lite-apiserver can use different cache storage strategies to achieve better results. In Light Edge, lite-apiserver uses file storage caching to reduce its own overhead and improve versatility. In Heavy Edge, lite-apiserver can use KV storage to improve read and write performance.

Next, we will introduce how to achieve the above functional requirements and design features from the architecture and key technologies of lite-apiserver.

Lite-apiserver Architecture and key Technology Architecture

The lite-apiserver architecture is shown in the figure

On the whole, lite-apiserver initiates a HTTPS Server to accept all Client requests (https request), and selects the corresponding ReverseProxy according to the Common Name in the request tls certificate (if the request does not have a mtls certificate, then use default) to forward the request to kube-apiserver. When the cloud edge network is normal, the corresponding https response is returned to client, and the response is stored asynchronously in the cache as needed; when the cloud edge is disconnected, the access to kube-apiserver timed out, and the cached data obtained from the cache is returned to client to achieve the goal of edge autonomy.

The port on which HTTPS Server listens on localhost (51003 in SurperEdge) accepts Https requests from Client.

Cert Mgr & & Transport Mgr Cert Mgr is responsible for managing TLS client certificates that connect to kube-apiserver. It periodically loads the configured TLS certificate and notifies Transport Mgr to create or update the corresponding transport if there is an update. Transport Mgr is responsible for managing transport. It receives notifications from Cert Mgr, creates a new transport, or closes the old connection to the transport whose certificate has been updated.

Proxy selects the corresponding ReverseProxy based on the Common Name in the request mtls certificate (if the request does not have a mtls certificate, it uses default) to forward the request to kube-apiserver. If the request succeeds, the result is returned directly to Client and Cache Mgr is called to cache the data; if the request fails, the data is read from Cache Mgr to Client.

Cache Mgr caches the result data of Get and List respectively according to the type of Client, and updates the corresponding List data according to the returned value of Watch.

Key technology 1. HTTPS Server

Under the current architecture, lite-apiserver only handles all requests from this node, and HTTP Server can be used to meet performance and security requirements. However, most components and business containers use client-go library to access kube-apiserver. If the authentication and authentication information of HTTP Server,Client is lost, it does not meet the requirements of permission management. Therefore, HTTPS Server must be adopted. Lite-apiserver 's TLS Server certificate needs to be issued with the same CA as kube-apiserver 's Server certificate.

two。 Support InCluster access

In general, Client accesses kube-apiserver through the URL of the specified kube-apiserver. When using lite-apiserver, you only need to replace the URL of the original kube-apiserver with the address of lite-apiserver. The recommended way to access kube-apiserver in Pod is through the DNS name kubernetes.default.svc, which will be resolved to the service IP, and then the service IP will be routed to kube-apiserver. Using lite-apiserver in this scenario requires a little "magic". In SuperEdge, application-grid-wrapper is deployed on each edge node in the form of DaemonSet, and the purpose of accessing the closed loop in the area is achieved by returning only the endpoints in the region to the kube-proxy. Taking advantage of this feature, application-grid-wrapper changes the endpoint of the Service kubernetes to the address of lite-apiserver and returns it to the node kube-proxy to support InCluster access.

3. Support for Bootstrap Token and certificate rotation of Client

Lite-apiserver uses Client's own authentication and authentication methods to access kube-apiserver on the cloud. For static token, bootstrap token, service account, etc., lite-apiserver only needs to transmit through the authentication and authentication information contained in the Header of Http Request. For the authentication method of TLS client certificate, lite-apiserver loads all the TLS client certificates used by Client by reading the configuration file, and uses these certificates to construct the corresponding HTTPS request kube-apiserver. In order to support Client's Bootstrap Token and certificate rotation, lite-apiserver needs to load and update these certificates periodically. The default time for certificates issued by kube-controller-manager is 1 year, and the period for lite-apiserver to load TLS client certificates should not be too short. However, if the certificate loading cycle is too long, there will be a problem that the certificate is not updated in time in the scenario where kubelet uses Bootstrap Token. To deal with these scenarios, lite-apiserver adopts an "elegant" certificate loading strategy: when there is an error in loading a certificate or the certificate expires, it enters the fast loading mode with a period of 1s; when the certificate is loaded successfully, it enters the normal loading mode with a period of 30min. When the certificate is updated, lite-apiserver uses the closeAll method provided by client-go to close the existing connection in case authentication fails.

4. Cache parsing and updating

Lite-apiserver needs to support caching of all types of resources, and cache parsing and updating is one of the keys to lite-apiserver implementation. Lite-apiserver caches each Client's Get and List requests for resources respectively, which results in a waste of storage space, but also avoids complex resource version conversion. For the request result of Watch type, lite-apiserver uses unstructured.UnstructuredJSONScheme to parse the meta information of the resource, and then updates the corresponding List data.

This is the end of the article on "how to achieve lite-apiserver". Thank you for reading! I believe that everyone has a certain understanding of the knowledge of "how to achieve lite-apiserver". If you want to learn more knowledge, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.