Shulou(Shulou.com)06/01 Report--

This article mainly explains "how to install edge K8s cluster and native K8s cluster with one click of edgeadm". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn "how to use edgeadm to install edge K8s cluster and native K8s cluster"!

The threshold is too high.

Users need to prepare a Kubernetes cluster in advance. For many users, the threshold is too high, and the building process is complex and easy to fail, which shuts out many people who want to use marginal capabilities.

Too restrictive.

A specific version of Kubernetes cluster built by specific tools is often required, and its versatility is too poor, and users are too restrictive to use it in the actual production environment.

It is troublesome to add edge nodes.

To add edge nodes, you need to rely on the tools of building Kubernetes cluster itself to add native Kubernetes nodes for transformation, which is highly dependent on third-party tools, and the operation process is troublesome and easy to make mistakes.

Poor automation ability

Whether building Kubernetes clusters or adding edge nodes, it is difficult to automate in the production environment, and the related processes still need to be redeveloped by their own team, so it is difficult to integrate.

In order to lower the threshold for users to experience edge capabilities, students in the cloud native community plan to develop a way to deploy edge Kubernetes clusters with one click, so that users can experience edge Kubernetes clusters more easily and easily.

Architecture design

In view of the above problems, in order to lower the threshold for users to use edge Kubernetes clusters and make edge Kubernetes clusters have production capacity, we have designed a solution that can deploy an edge Kubernetes cluster with one click, which completely shields the installation details and allows users to experience edge capabilities with zero threshold.

Original intention

Allows users to use edge Kubernetes clusters easily and without a threshold, and can really use edge capabilities in a production environment

target

One-click use

Can build an edge Kubernetes cluster with one click

It is very simple and flexible to add edge nodes with one button.

Two kinds of installation to create scenery

Support for online installation

Support for offline installation, making the privatized environment easy

Can be produced and used

Do not encapsulate too much, so that teams who want to use edge Kubernetes clusters can easily integrate their internal systems and produce availability.

Zero learning cost

Try to be consistent with the way kubeadm is used, so that users have no additional learning costs. If they can use kubeadm, they will use edgeadm.

Principle

Do not modify the kubeadm source code

Try to quote and reuse the source code of kubeadm, and try not to modify the source code of kubeadm, so as to avoid the hidden danger of later upgrade.

Based on kubeadm but higher than kubeadm, it does not need to be limited by the design of kubeadm, as long as it makes it easier for users to use.

Allows users to choose whether or not to deploy edge capability components

Allows users to customize the configuration of edge capability components

Design and implementation

We studied the source code of kubeadm and found that we can use kubeadm to create native Kubernetes clusters, join nodes, and workflow ideas to deploy edge Kubernetes clusters with one click, and you can perform the installation steps step by step. This is the deployment solution that we want to be simple, flexible and low learning cost. So we stand on the shoulders of giants, using the idea of Kubedam, reuse the source code of kubeadm, and design the following solution.

The kubeadm init cluster/join node part completely reuses the source code of kubadm, and all the logic is exactly the same as kubeadm.

This scheme has the following advantages:

Fully compatible with kubeadm

We just stood on the shoulder of kubeadm, set some configuration parameters needed by the edge cluster before kubeadm init/join, automated the initialization of Master or Node nodes, and installed the container runtime. After the kubeadm init/join is completed, the CNI network plug-in is installed and the corresponding edge capability components are deployed.

We have quoted the kubeadm source code in the way of Go Mod, and the source code of kubeadm has not been modified one line in the whole process, which is completely native, which is ready to upgrade a later version of kubeadm.

One-click, easy to use, flexible and automatic

Edgeadm init clusters and join nodes completely retain the original parameters and processes of kubeadm init/join, but automatically initialize the nodes and install the container runtime. You can use the edgeadm-- enable-edge=fasle parameter to install the native Kubernetes cluster, or you can use the edgeadm-- enable-edge=true parameter to install the edge Kubernetes cluster.

You can Join any node where the Kube-apiserver is located, or you can join master. Join master also continues the way of kubeadm. To build highly available nodes, you can directly use join master to expand Master nodes when needed to achieve high availability.

No learning cost, exactly the same as using kubeadm

Because kubeadm init cluster/join node partially reuses the source code of kubadm completely, all the logic is exactly the same as kubeadm, and completely retains the usage of kubeadm and all flag parameters, and the usage is exactly the same as that of kubeadm, without any new learning costs. Users can customize edge Kubernetes clusters by kubeadm parameters or using kubeadm.config.

Edge node security enhancement

With the help of Kubernetes Node authentication mechanism, we enable the NodeRestriction access plug-in by default to ensure that each node has a unique identity and only a minimum permission set, and cannot operate other edge nodes even if one edge node is compromised.

Kubelet We also enable the Kubelet configuration certificate rotation mechanism by default. When the Kubelet certificate is about to expire, a new key is automatically generated and a new certificate is requested from Kubernetes API. Once the new certificate is available, it will be used for connection authentication with Kubernetes API.

Install Edge Kubernetes Cluster with edgeadm

The following process community has recorded a detailed tutorial video, which can be installed in conjunction with documentation:

Video resource link: install edge Kubernetes cluster and native Kubernetes cluster with one click of edgeadm

Installation condition

In compliance with the minimum requirements of kubeadm, Master & & Node has a minimum 2C2G and disk space of not less than 1G

Currently, amd64 and arm64 are supported.

Other systems can compile edgeadm and make corresponding system installation packages, please refer to 5. 5. Customize the Kubernetes static installation package

Supported Kubernetes version: greater than or equal to v1.18. The installation package provided is only Kubernetes v1.18.2.

For other Kubernetes versions, please refer to 5. 0. Custom Kubernetes static installation package, self-made.

Download the edgeadm static installation package

Download the edgeadm static installation package and copy it to all Master and Node nodes.

Note that the "arch=amd64" parameter is modified. [amd64, amd64] is currently supported. Download the architecture of your own machine. Other parameters remain unchanged.

Arch=amd64 version=v0.3.0-beta.0 & & rm-rf edgeadm-linux-* & & wget https://superedge-1253687700.cos.ap-guangzhou.myqcloud.com/$version/$arch/edgeadm-linux-$arch-$version.tgz & & tar-xzvf edgeadm-linux-* & & cd edgeadm-linux-$arch-$version &. / edgeadm

The installation package is about 200m. For more information about the installation package, please see 5. 5. Customize the Kubernetes static installation package.

If it is slow to download the installation package, you can directly view the corresponding version of SuperEdge, download edgeadm-linux-amd64/arm64-*.0.tgz, and extract it.

The one-click install edge-independent Kubernetes clustering feature is supported from SuperEdge-v0.3.0-beta.0. Please download v0.3.0-beta.0 and later versions.

Install the edge Kubernetes Master node. / edgeadm init-- kubernetes-version=1.18.2-- image-repository superedge.tencentcloudcr.com/superedge-- service-cidr=10.96.0.0/12-- pod-network-cidr=192.168.0.0/16-- install-pkg-path. / kube-linux-*.tar.gz-- apiserver-cert-extra-sans=-- apiserver-advertise-address=-- enable-edge=true-vault 6

Where:

-- enable-edge=true: whether to deploy edge capability components. Default is true.

-- enable-edge=false means to install a native Kubernetes cluster, which is exactly the same as the cluster built by kubeadm

-- install-pkg-path: the address of the Kubernetes static installation package

-- the value of install-pkg-path can be either the path on the machine or the network address (for example, http://xxx/xxx/kube-linux-arm64/amd64-*.tar.gz, can be secret-free wget). Be careful to use the Kubernetes static installation package that matches the machine system.

-- apiserver-cert-extra-sans: the certificate extension address of kube-apiserver

It is recommended to sign the public network IP or domain name of the Master node, as long as the IP or domain name of the signed Master node can be accessed by the edge node. Of course, the private network IP is also allowed, as long as the edge node can access the Kube-apiserver through this IP. If you customize the domain name, you can configure hosts on all Matser and Node nodes

Signing the IP and domain name of the public network is because the edge node is generally not in the same local area network as the Master node, so you need to join and access the Master through the external network.

-- image-repository: address of image repository

If superedge.tencentcloudcr.com/superedge is slow, you can switch to other accelerated image repositories, as long as you can Pull kube-apiserver,kube-controller-manager,kube-scheduler,kube-proxy,etcd, pause... A mirror image is fine.

Other parameters have exactly the same meaning as kubeadm and can be configured according to the requirements of kubeadm.

You can also use kubeadm.config to configure the original parameters of kubeadm and create an edge Kubernetes cluster through edgeadm init-- config kubeadm.config-- install-pkg-path. / kube-linux-*.tar.gz.

If there is no problem in the execution process, and the cluster is initialized successfully, the following content will be output:

Your Kubernetes control-plane has initialized successfully!To start using your cluster You need to run the following as a regular user: mkdir-p $HOME/.kube sudo cp-I / etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id-u): $(id-g) $HOME/.kube/config You should now deploy a podnetwork to the cluster.Run "kubectl apply-f [podnetwork] .yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running The following on each as root:edgeadm join xxx.xxx.xxx.xxx:xxx-- token xxxx\-- discovery-token-ca-cert-hash sha256:xxxxxxxxxx-- install-pkg-path

If a problem occurs during execution, the corresponding error message will be returned directly, and the initialization of the cluster will be interrupted. You can use the. / edgeadm reset command to roll back the initialization operation of the cluster.

Set up the Master kube-config file

To enable non-root users to run kubectl, run the following command, which is also part of the edgeadm init output:

Mkdir-p $HOME/.kubesudo cp-I / etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id-u): $(id-g) $HOME/.kube/config

Or, if you are a root user, you can run:

Export KUBECONFIG=/etc/kubernetes/admin.conf

Note that you save the. / edgeadm join command from the. / edgeadm init output, which will be used later when you add the Node node.

The validity period of token is the same as that of kubeadm for 24 hours. After expiration, you can use. / edgeadm token create to create a new token.

The value generation of discovery-token-ca-cert-hash is also the same as kubeadm, which can be generated by executing the following command on the Master node.

Openssl x509-pubkey-in / etc/kubernetes/pki/ca.crt | openssl rsa-pubin-outform der 2 > / dev/null | openssl dgst-sha256-hex | sed's / ^. * / / 'Join edge node

Execute on the edge node. Download the edgeadm static installation package, or upload the edgeadm static installation package to the edge node by other means, and then execute the following command:

/ edgeadm join: Port-- token xxxx\-- discovery-token-ca-cert-hash sha256:xxxxxxxxxx-- install-pkg-path-- enable-edge=true

Where:

Port is the address where the node accesses the Kube-apiserver service

The address of the Kube-apiserver service prompted by the edgeadm init joining node can be changed to the Master node external network IP/Master node private network IP/ domain name, depending on whether you want the node to access the Kube-apiserver service through the public network or the private network.

-- enable-edge=true: whether the joined node is an edge node (whether to deploy edge capability components). Default true

-- enable-edge=false represents the join native Kubernetes cluster node, which is exactly the same as the kubeadm join node

If there is no problem during execution, and the new Node successfully joins the cluster, the following will be output:

This node has joined the cluster:* Certificate signing request was sent to apiserver and a response was received.* The Kubelet was informed of the new secure connection details.Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

If a problem occurs during the execution, the corresponding error message will be returned directly, and the addition of the node will be interrupted. You can use the. / edgeadm reset command to roll back the operation of joining the node and re-join.

Tip: if the edge node join is successful, it will call a label: superedge.io/edge-node=enable to the edge node to facilitate subsequent applications to select the application scheduling to the edge node with nodeSelector.

Native Kubernetes nodes, like kubeadm's join, do nothing.

Use edgeadm installation edge high availability Kubernetes cluster installation prerequisite

Prepare a Master VIP as a unified entry for available load balancers

3 machines that meet the minimum requirements of kubeadm as Master nodes

Three machines that meet the minimum requirements of kubeadm do worker nodes.

Install Haproxy

Install Haproxy as the cluster master entry on Master

Note: replace the

# yum install-y haproxy# cat / etc/haproxy/haproxy.cfgglobal log 127.0.0.1 local2 chroot / var/lib/haproxy pidfile / var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon stats socket / var/lib/haproxy/statsdefaults mode httplog global option httplog Option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000frontend main *: 5000 acl url_static path_beg-I / static / images / javascript / stylesheets acl url_static path_end-I .jpg .gif .png .css .js use_backend static if url_static default_backend appfrontend kubernetes-apiserver mode tcp bind *: 16443 option tcplog default _ backend kubernetes-apiserverbackend kubernetes-apiserver mode tcp balance roundrobin server master-0: 6443 check # here replace Master VIP for the user's own VIPbackend static balance roundrobin server static 127.0.0.1 Master VIP 4331 checkbackend app balance roundrobin server app1 127.0.0.1 Master VIP 5001 check server app2 127.0.0.1 backend kubernetes-apiserverbackend kubernetes-apiserver mode tcp balance roundrobin server master-0 5002 check server app3 127.0.1 Master VIP 5003 check server app4 127.0.0.1 Master VIP 5004 checkEOF installation Keepalived

Install Keepalived on all Master and do the same: note:

Replace the

In the following keepalived.conf configuration file

And

The configuration of different Master needs to change position, do not enter it incorrectly.

# # install keepalived yum install-y keepalived cat / etc/keepalived/keepalived.conf! Configuration File for keepalived global_defs {smtp_connect_timeout 30 router_id LVS_DEVEL_EDGE_1} vrrp_script checkhaproxy {script "/ etc/keepalived/do_sth.sh" interval 5} vrrp_instance VI_1 {state BACKUP interface eth0 nopreempt virtual_router_id 51 priority 100 advert_int 1 authentication {auth_type PASS auth_pass aaa} Virtual_ipaddress {# here replace Master VIP with the user's own VIP} unicast_src_ip unicast_peer {} notify_master "/ etc/keepalived/notify_action.sh master" notify_backup "/ etc/keepalived/notify_action.sh BACKUP" notify_fault "/ etc/keepalived/notify_action.sh FAULT" notify_stop "/ etc/keepalived/notify_ Action.sh STOP "garp_master_delay 1 garp_master_refresh 5 track_interface {eth0} track_script {checkhaproxy}} EOF install highly available Edge Kubernetes Master

Perform cluster initialization in one of the Master

. / edgeadm init-- control-plane-endpoint-- upload-certs-- kubernetes-version=1.18.2-- image-repository superedge.tencentcloudcr.com/superedge-- service-cidr=10.96.0.0/12-- pod-network-cidr=192.168.0.0/16-- apiserver-cert-extra-sans=-- install-pkg-path-vault 6

The meaning of the parameter is the same as 3. Use edgeadm to install the edge Kubernetes cluster, others are the same as kubeadm, which is not explained here

If there is no problem in the execution process, and the cluster is initialized successfully, the following content will be output:

Your Kubernetes control-plane has initialized successfully!To start using your cluster You need to run the following as a regular user: mkdir-p $HOME/.kube sudo cp-I / etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id-u): $(id-g) $HOME/.kube/config You should now deploy a podnetwork to the cluster.Run "kubectl apply-f [podnetwork] .yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of the control-plane Node running the following command on each as root: edgeadm join xxx.xxx.xxx.xxx:xxx-token xxxx\-discovery-token-ca-cert-hash sha256:xxxxxxxxxx\-control-plane-certificate-key xxxxxxxxxx-install-pkg-path Please note that the certificate-key gives access to cluster sensitive data Keep it secret!As a safeguard, uploaded-certs will be deleted in two hours If necessary, you can use "edgeadm init phase upload-certs-upload-certs" to reload certs afterward.Then you can join any number of worker nodes by running the following on each as root:edgeadm join xxx.xxx.xxx.xxx:xxxx-- token xxxx\-- discovery-token-ca-cert-hash sha256:xxxxxxxxxx-- install-pkg-path

If a problem occurs during execution, the corresponding error message will be returned directly, and the initialization of the cluster will be interrupted, and the initialization operation of the cluster will be rolled back using the. / edgeadm reset command.

To enable non-root users to run kubectl, run the following command, which is also part of the edgeadm init output:

Mkdir-p $HOME/.kubesudo cp-I / etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id-u): $(id-g) $HOME/.kube/config

Or, if you are a root user, you can run:

Export KUBECONFIG=/etc/kubernetes/admin.conf

Note that you need to save the. / edgeadm join command of the. / edgeadm init output, which is needed to add Master nodes and edge nodes.

Join Master node

Execute the. / edgeadm join command on another Master

. / edgeadm join xxx.xxx.xxx.xxx:xxx-- token xxxx\-discovery-token-ca-cert-hash sha256:xxxxxxxxxx\-- control-plane-- certificate-key xxxxxxxxxx\-- install-pkg-path

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.