Shulou(Shulou.com)06/01 Report--

This article introduces you how to use helm-controller to simplify the development of container cloud platform application store, the content is very detailed, interested friends can refer to, I hope it can be helpful to you.

To read skillfully and develop accordingly, you probably need to know the following knowledge:

The K8s rest api model needs to be understood.

Http protocol needs to be mastered.

Understanding of openapi specification.

Proficient in linux.

Understanding of the theoretical framework of helm.

K8s crd understanding, controller concept mastery, rbac permission model understanding.

Understanding and use of index.yaml, a private library of helm.

Configuration and use of core dns.

Overall structure diagram

What is helm?

Helm is the package manager of K8s and the de facto standard for complex application deployment on K8s platform. Includes application packaging, deployment, upgrade, rollback, uninstall and other lifecycle management functions.

Architecture change

The upgrade of helm from v2 to v3 removes an important component, tiller, and the overall architecture is more concise.

The deficiency of helm Architecture in the Development of Cloud Management platform

Helm so far, there is still no official ga version of api. Chart download, deployment, upgrade, uninstall, all rely on cli. In the multi-cluster environment, it is difficult for cli to meet the business requirements of the platform.

By looking at github issue, the community probably has two solutions:

Encapsulate cli into api. This approach still exists that each cluster needs to deploy helm binaries to the master node through ssh or ansible, adding a burden to the underlying deployment work.

CRD . The core capabilities of helm are packaged into docker images and deployed in a K8s cluster to provide capabilities in the form of controller. Use crd to complete release deployment, uninstall, upgrade, rollback and other business actions.

The biggest problem with cli is that it does not conform to the idea of cloud native, and the cli method and helm version are locked. If you want to upgrade helm, you need to re-adapt to parse console content. The problem with crd is that there is no official ga yet. But still look forward to the controller way.

Our team initially tried the first method, but the results were not satisfactory. Coincidentally, it was discovered at that time that Finch Cloud had opened up helm v3 controller captain, so a second attempt was made based on community captain, and the functional development was finally completed.

At that time, I searched helm controller on github and found two warehouses, one provided by controller provided by rancher and the other provided by Finch Cloud. After a simple test, captain was installed and tested successfully at one time, and combined with internal discussion, finally decided to develop based on captain.

Captain

Github: https://github.com/alauda/captain

Introduction

The Helm 3 Design Proposal exists for a while and currently it is still under heavy development. Captain comes as the first implementation of Helm v3 Controller based on the Proposal. This project is based on the core helm v3 code, acting as a library. Since it's not officially released yet (alpha stage for now), some modifications were made to help implement this controller on a fork: alauda/helm (will be deprecated once Helm's library is released).

Captain is the open source helm v3 controller of Finch Cloud. It is internally dependent on helm library. So the core logic is consistent with helm client. After the official ga of helm later, you can migrate back to the official version, which is so easy for interface-oriented programming of java.

Open source based on Apache 2.0 protocol

Captain deploys helm as deployment internally.

Quick installation test

Installation steps:

Kubectl create ns captain-systemkubectl create clusterrolebinding captain--serviceaccount=captain-system:default-- clusterrole=cluster-adminkubectl apply-n captain-system-f https://raw.githubusercontent.com/alauda/captain/master/artifacts/all/deploy.yaml

Uninstall:

Kubectl delete-n captain-system-f https://raw.githubusercontent.com/alauda/captain/master/artifacts/all/deploy.yamlkubectl delete ns captain-system

Install nginx chart

Kind: HelmRequestapiVersion: app.alauda.io/v1alpha1metadata: name: nginx-ingressspec: chart: stable/nginx-ingress

View deployment results

Root@VM-16-12-ubuntu:~/demo# kubectl get podsNAME READY STATUS RESTARTS AGEnginx-ingress-controller-57987f445c-9rhv5 1 16sroot@VM-16 1 Running 0 16snginx-ingress-default-backend-7679dbd5c9-wkkss 1 Running 0 16sroot@VM-16-12-ubuntu:~/demo# kubectl get hrNAME CHART VERSION NAMESPACE ALLCLUSTER PHASE AGEnginx-ingress stable/nginx-ingress default Synced 23schart repo problem

Captain comes with the official helm repository of stable by default. There is no problem with the official repository address of helm. However, if a walled docker image is used in the chart image, it cannot be downloaded. The warehouse address https://developer.aliyun.com/hub/ provided by aliyun is used in the test. Only in this way can captain controller download the chart image successfully.

When the test is finished, we need to connect K8s with the private chart library of the private network, and we need to create a new yaml file of ChartRepo

ApiVersion: app.alauda.io/v1alpha1kind: ChartRepometadata: name: cloud namespace: captain-systemspec: url: https://harbor.offline/chartrepo/library

Then use kubectl create-f fileName to add to k8s. It should be noted that we use harbor to manage docker image and helm image. Because of the problem of docker, we use a self-signed certificate. Captain will verify the certificate when synchronizing according to the address. We have also communicated with the authorities and solved this problem. At present, captain has been ga and can be used directly without worrying about the certificate.

RBAC permission problem

In the management of cloud platform, we use k8s api through servicecount. When captain is installed, unlike using user account on the command line, we need an extra step to add permissions.

Kubectl create clusterrolebinding default-serviceaccount=default:default-clusterrole=cluster-admin

Captain sdk problem

Captain officially only provides sdk for go and python. Based on this, we definitely want to encapsulate a java sdk for captain.

At the bottom of the architecture, we use the official sdk of K8s for development.

Based on the simple understanding of crd and K8s openapi, combined with official instructions, we attempted to generate sdk, but failed. See issue for details. We also contacted the author and learned that captain did not do verification based on schema, but internally used webhook for verification. Based on this background, there is no way to generate sdk directly using openapi specification. Later, we directly use kubectl-v9 to verify the message and develop the code.

When you use the kubectl command line to perform any operation, append the-v9 parameter to get detailed http message information.

Root@master:/home/kylin# kubectl get pod-v9I0414 22 root@master:/home/kylin# kubectl get pod 42 loader.go:359 53.981748 16582 loader.go:359] Config loaded from file: / root/.kube/configI0414 22 14 42173 16582 round_trippers.go:419] curl-k-v-XGET-H "Accept: application/json;as=Table;v=v1beta1 G=meta.k8s.io Application/json "- H" User-Agent: kubectl/v1.15.5 (linux/amd64) kubernetes/20c265f "'https://192.168.4.139:6443/api/v1/namespaces/default/pods?limit=500'I0414 22 https://192.168.4.139:6443/api/v1/namespaces/default/pods?limit=500'I0414 22 User-Agent 54.077898 16582 round_trippers.go:438] GET https://192.168.4.139:6443/api/v1/namespaces/default/pods?limit=500 200 OK in 35 millisecondsI0414 22 round_trippers.go: 444] Response Headers:I0414 22:42:54.078006 16582 round_trippers.go:447] Content-Type: application/jsonI0414 22:42:54.078054 16582 round_trippers.go:447] Date: Tue 14 Apr 2020 14:42:54 GMTI0414 22 GMTI0414 42 request.go:947 54.078394 16582 request.go:947] Response Body: {"kind": "Table", "apiVersion": "meta.k8s.io/v1beta1", "metadata": {"selfLink": "/ api/v1/namespaces/default/pods", "resourceVersion": "14332801"}, "columnDefinitions": the full message is too long Omit it! No kind "Table" is registered for version "meta.k8s.io/v1beta1" in scheme "k8s.io/kubernetes/pkg/api/legacyscheme/scheme.go:30" NAME READY STATUS RESTARTS AGEbusybox 1 Running 970 39dnginx-1585049022-b4f4c56c9-dvspz 1/1 Running 24 12dnginx-deployment-5bd886c88c-28d6q 0/1 Pending 0 2d1hnginx-deployment-5bd886c88c-968pd 0/1 MatchNodeSelector 0 4d3hnginx-deployment-5bd886c88c-dnh8q 0/1 MatchNodeSelector 0 4d3hnginx-deployment-5bd886c88c-pk9xz 0/1 Pending 0 2d1h

Combined with the CustomObjectsApi provided by K8s official java sdk. It is relatively easy to develop a set of interfaces corresponding to the life cycle of chart images.

Deploy var customObjectsApi = new CustomObjectsApi (apiClient) Var json = new JsonObjectBuilder () .set ("apiVersion", "app.alauda.io/v1alpha1") .set ("kind", "HelmRequest") .set ("metadata", new JsonObjectBuilder (). Set ("name", name). Build () .set ("spec", new JsonObjectBuilder () .set ("chart", chart) .set ("namespace") Namespace) .set ("releaseName", name) .set ("values", Map2JsonUtil.map2Json (params)) .set ("version", version)) .build () CustomObjectsApi.createNamespacedCustomObject ("app.alauda.io", "v1alpha1", "default", "helmrequests", json, null); uninstall var customObjectsApi = new CustomObjectsApi (apiClient); customObjectsApi.deleteNamespacedCustomObject ("app.alauda.io", "v1alpha1", "default", "helmrequests", "test-nginx", new V1DeleteOptions (). GracePeriodSeconds (0L) .originationPolicy ("Foreground"), null, null, null); upgrade

Here you can choose to play patch or direct replace, which is consistent with the concept of K8s.

Roll back

Captain does not provide rollback support as native as deployment. You need to save the parameters of each installation or upgrade externally, and then re-replace the parameters of the specified version to simulate the rollback.

Other instructions

Overall, it took us three weeks to complete the development of the first version of the app store and the joint adjustment of the page interface. This is much faster than expected with cli, and we only need to add two additional lines to our ansbile deployment script to install captain.

Using private helm repo, by default, the coredns in the cluster forwards non-cluster addresses to the local / etc/resolv.conf. At this time, make sure that the / etc/resolv.confdns address of the K8s host is changed to the dns server address of the private network. Otherwise, captain controller cannot find the private helm repo, and the error is timeout.

In the process of development, if you encounter a problem that cannot be located, you can directly check the log of captain-controller to deal with it.

For network problems, it is best to deploy a busybox with built-in tools such as nslookup,wget. It is convenient for network detection.

ApiVersion: v1 kind: Pod metadata: name: busybox namespace: default spec: containers:-name: busybox image: busybox:1.28.4 command:-sleep-"3600" imagePullPolicy: IfNotPresent restartPolicy: Always

Use kubectl create-f busybox.yaml to complete busybox deployment, use kubectl exec-it busybox sh to enter the container, and use nslookup,wget for network detection.

This is the end of the development of the application store on how to use helm-controller to simplify the container cloud platform. I hope the above content can be of some help and learn more. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.