Shulou(Shulou.com)05/31 Report--

This article will explain in detail how the K8s platform deals with Pod pre-authorization. The content of the article is of high quality, so the editor shares it for you as a reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

Background

* * how to pre-authorize new devices when new services are deployed or expanded? * * I believe you are no stranger to this problem. For security reasons, important components and storage within the company often control the source of access requests, such as IP access authorization of CDB, module authorization of OIDB and VASKEY command words, and so on. They either have their own authorized WEB that allows users to apply for bills of lading, or provide authorized API that can be called by the operation and maintenance platform. However, when discovering registration, routing systems often need to accurately obtain the regional information of IP devices to provide the ability to access the nearest, which requires pre-registration of CMDB.

In the past, when using CVM/TVM to deploy business, this problem can be easily dealt with, because we have got a virtual machine in advance, assigned IP and registered CMDB. What the business needs to do is to use this IP to authorize the bill of lading, deploy business programs, and add routing online after everything is complete. This process can be automated with the pipelining capabilities of the operation and maintenance platform.

Different from VM's step-by-step deployment after obtaining available devices, Kubernetes manages the entire lifecycle of Pod from production, IP allocation, business container startup, and route maintenance, and is automatically managed by the Control Loop of multiple system Controller. Image-based deployment provides the guarantee of scalability and consistency of business instances, the destruction and reconstruction of Pod becomes normal, and IP cannot be fixed.

Business is often faced with a variety of pre-authorization needs, the average authorization time varies from seconds to a few minutes, most of the authorization API is not designed to carry high QPS, there is a certain degree of complexity. We need to find a way to process the authorization before the business container gets up after the Pod IP allocation, block and guarantee the success of the subsequent process, and control the pressure of the reconstruction process on the authorized API.

After design and iterative optimization, the TKEx-CSIG platform provides the authorization capability of easy-to-use products, which is convenient to deal with this kind of Pod pre-authorization problem.

Architecture and competency resolution architecture

The above figure shows the architecture of the authorization system. The core idea is to use the feature that init Container executes before the business container to implement complex logical preprocessing before the start of the business Pod. The official definition of init Container is as follows

This page provides an overview of init containers: specialized containers that run before app containers in a Pod. Init containers can contain utilities or setup scripts not present in an app image

If it is a small-scale or individual business solution, we can simply inject init Container into the business Worklooad yaml and call the required authorized API implementation. To achieve the platform production capability, we also need to consider the following points:

Easy to use and maintainable

It is necessary to fully consider the efficiency and manageability of business use, and manage permissions as a resource by platform records to reduce the intrusive impact of changes on the business.

Frequency limitation and self-healing

Permission API is often not designed for high QPS and needs to restrict calls to protect the downstream.

Authority convergence

Security. Destruction and reconstruction of Pod may lead to changes in IP. Consider actively reclaiming expired permissions.

Product capability of authorization process

The business only needs to register the required permission resources on the platform WEB console, configure the permission group, and associate the permission group to the Workload. The platform automatically injects the init Container configuration, transfers the authorization configuration index and related information through the ENV, and carries out the authorization process when the Pod is created. The functional designs of several components involved in the authorization process are as follows:

Init-action-client

Init Container, only make a trigger device, only do one thing, that is, initiate a HTTP call request and keep it immutable, so that when the function is iterated, there is no need to modify the yaml of the business, and the main logic moves backward.

Init-action-server

Deployment deployment can be scaled out, perform preprocessing logic, pre-register CMDB and other operations, and initiate pipeline calls, start the permission application process and poll query, and expose the process information associated with POD to facilitate business self-inspection and administrator positioning problems. The Backoff retry and circuit breaker logic mentioned later are also implemented here.

PermissionCenter

The platform management and control component, located outside the cluster, is responsible for the storage and actual application of permission resources. It includes a permission resource center, which stores the permission details parameters of business registration for convenient reuse, provides authority Set group management, simplifies the parameter transfer in the authorization process, and uses the producer / consumer mode to realize the authorization API call and result query based on Pipline.

Circuit breaker and Backoff retry mechanism

There may be a lot of abnormal conditions in the authorization process, such as misconfiguration of permission parameters, degradation or unavailability of authorization API service, and even interface errors and timeouts caused by network reasons. Authorized API is often not designed to support high QPS, so we use timeout retry, circuit breaker and exponential Backoff retry to make a fault tolerance.

Time-out retry

This is reflected in the timeout setting and retry mechanism of API calls and asynchronous tasks. In response to instantaneous failures, abnormal exit of the init-action-client container will also be rebuilt, and each creation is a new round of retry.

Circuit breaker

A Configmap is used to record the number of failed applications for Pod permission in the cluster. If the application is disconnected for 3 times, the application will not be granted. And provide a reset capability, exposed to the front end, so that users and administrators can easily retry.

Exponential Backoff

Circuit breaker mode can block cases such as user configuration errors that can never be granted successfully, but cannot cope with long-term transient failures. For example, during the termination period, the authorization API backend may have a period of denial of service for a period of 10 minutes to a few hours, when a large number of Pod authorization hits the circuit breaker rules can not continue to authorize, artificial processing is not timely and cumbersome. We add an exponential Backoff with jitter for each Pod and record the most recent failure timestamp, which allows one attempt after a certain period of time. If successful, reset the Backoff to the specified Pod. If the timestamp is not updated successfully, the parameters are as follows

Bk: = & PodBreaker {NamespacePod: namespacePod, LastRequestFailTime: time.Now (), Backoff: wait.Backoff {Duration: 2 * time.Minute, Factor: 2.0, Jitter: 1.0, Steps: 5, Cap: 1 * time.Hour,},} Finalizer Convergence permission

The convergence of permissions is often ignored, but security needs to be considered. The destruction and reconstruction of Pod may be normal, and IP may change dynamically, resulting in a large number of junk permissions over a long period of time, or authorized IP may be assigned to other business Pod, resulting in security risks. We have made a Finalizer controller to reclaim permissions before Pod termination. The reclaim action is idempotent and best-effort, because the ability to reclaim also depends on whether the permission party has the ability to reclaim. We will consider this point for new interfacing permissions, such as Tencent Cloud MySQL IP automatic authorization.

In order to reduce the number of Finalizer calls and avoid affecting unauthorized Pod as much as possible, we only identify the Finalizer tag on the Pod,Patch with authorized init Container when the Pod makes a change event, reclaim the permission and delete the Finalizer when the Pod is reduced and terminated, and then the GC will delete the Pod.

Kind: Podmetadata: annotations:~ creationTimestamp: "2020-11-13T09:16:52Z" finalizers:-stke.io/podpermission-protection is here about how the K8s platform deals with Pod pre-licensing. I hope the above content can be helpful to you and learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.