Shulou(Shulou.com)06/03 Report--

Rancher official Chinese document: https://www.rancher.cn/docs/rancher/v2.x/cn/overview/

This experiment is for the deployment of Ranch seven-tier load balancer Helm HA

1. Environment preparation 1. Server configuration

The server is configured with Tencent Cloud virtual machines as follows:

Host name system version private network ip public network ip configuration master1CentOS 7.6172.27.100.101xxx4C 16Gmaster2CentOS 7.6172.27.100.101xxx4C 16Gmaster3CentOS 7.6172.27.100.101xxx4C 16G

Because layer-7 load balancer is used, a LB is required. Tencent Cloud cloud load balancer is used here (nginx can be used in the experimental environment)

(1) configure hosts (/ etc/hosts) for each host and add host_ip $hostname to the / etc/hosts file.

(2) close selinux:

Sudo sed-I's etc/selinux/config SelinuxplenforcingAccording to SELINUXAfter

(3) disable the firewall or open the corresponding port in the test environment, refer to https://www.rancher.cn/docs/rancher/v2.x/cn/install-prepare/references/

Note: my private network is all enabled here, but an error was reported during deployment. You need to open the corresponding port so that the public network ip of the server can be accessed ~ (I don't know what happened if I didn't use the private network ip).

2. Installation and configuration of Docker

When installing Docker normally, the only thing to note is that the root account cannot be used when installing K8S cluster through RKE because of the security restrictions of CentOS. CentOS users use non-root users to transport docker.

(1) configure and install the source yum install ca-certificates; update-ca-trust;cp / etc/yum.repos.d/CentOS-Base.repo / etc/yum.repos.d/CentOS-Base.repo-bakcat / etc/yum.repos.d/CentOS-Base.repo# CentOS-Base.repo## The mirror system uses the connecting IP address of the client and the# update status of each mirror to pick mirrors that are updated to and# geographically close to the client. You should use this for CentOS updates# unless you are manually picking other mirrors.## If the mirrorlist= does not work for you As a fall back you can try the # remarked out baseurl= line instead.## [base] name=CentOS-$releasever-Base-mirrors.aliyun.comfailovermethod=prioritybaseurl= http://mirrors.aliyun.com/centos/$releasever/os/$basearch/ http://mirrors.aliyuncs.com/centos/$releasever/os/$basearch/ http://mirrors.cloud.aliyuncs.com/centos/$releasever/os/$basearch/gpgcheck=1gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7#released updates [updates ] name=CentOS-$releasever-Updates-mirrors.aliyun.comfailovermethod=prioritybaseurl= http://mirrors.aliyun.com/centos/$releasever/updates/$basearch/ http://mirrors.aliyuncs.com/centos/$releasever/updates/$basearch/ http://mirrors.cloud.aliyuncs.com/centos/$releasever/updates/$basearch/gpgcheck=1gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7#additional packages that may be [extras] name=CentOS-$releasever-Extras-mirrors.aliyun.comfailovermethod=prioritybaseurl=http:// Mirrors.aliyun.com/centos/$releasever/extras/$basearch/ http://mirrors.aliyuncs.com/centos/$releasever/extras/$basearch/ http://mirrors.cloud.aliyuncs.com/centos/$releasever/extras/$basearch/gpgcheck=1gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7#additional packages that extend functionality of existing packages [centosplus] name=CentOS-$releasever-Plus-mirrors.aliyun.comfailovermethod=prioritybaseurl= http://mirrors.aliyun.com/centos/$releasever/centosplus/$basearch / http://mirrors.aliyuncs.com/centos/$releasever/centosplus/$basearch/ http://mirrors.cloud.aliyuncs.com/centos/$releasever/centosplus/$basearch/gpgcheck=1enabled=0gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7#contrib-packages by Centos users [contrib] name=CentOS-$releasever-Contrib-mirrors.aliyun.comfailovermethod=prioritybaseurl= http://mirrors.aliyun.com/centos/$releasever/contrib/$basearch/ http://mirrors.aliyuncs.com/ Centos/$releasever/contrib/$basearch/ http://mirrors.cloud.aliyuncs.com/centos/$releasever/contrib/$basearch/gpgcheck=1enabled=0gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7EOF (2) install Docker# define username NEW_USER=rancher# add user (optional) sudo adduser $NEW_USER# set password for new user sudo passwd $NEW_USER# add sudo permission sudo echo "$NEW_USER ALL= (ALL) for new user ) ALL "> > / etc/sudoers# uninstalls the old version of Docker software sudo yum remove docker\ docker-client\ docker-client-latest\ docker-common\ docker-latest\ docker-latest-logrotate\ docker-logrotate\ docker-selinux\ docker-engine-selinux\ Docker-engine\ container*# defines the installation version export docker_version=19.03.1# step 1: install some necessary system tools sudo yum update-y Sudo yum install-y yum-utils device-mapper-persistent-data\ lvm2 bash-completion;# Step 2: add software source information sudo yum-config-manager-- add-repo\ http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo;# Step 3: update and install Docker-CEsudo yum makecache all;version=$ (yum list docker-ce.x86_64-- showduplicates | sort-r | grep ${docker_version} | awk'{print $2}') Sudo yum-y install-- setopt=obsoletes=0 docker-ce-$ {version} docker-ce-selinux-$ {version}; # if you have installed a higher version of Docker, you can downgrade the installation (optional) yum downgrade-- setopt=obsoletes=0-y docker-ce-$ {version} docker-ce-selinux-$ {version}; # add the current user to the docker group sudo usermod-aG docker $NEW_USER;# setting to boot sudo systemctl enable docker; (3) modify the Docker configuration

Daemon.json is located at / etc/docker/daemon.json by default, and if there is no manual creation, systemd-based systems are all in the same path. Changing the Docker configuration by modifying daemon.json is also the method officially recommended by Docker.

1. Configure private warehouse

By default, Docker only trusts the TLS encrypted repository address (https), and all non-https repositories cannot log in or pull images by default. Insecure-registries literally means unsecured warehouse, and credit is given to non-https repositories by adding this parameter. Multiple insecure-registries addresses can be set and written in an array, and the address cannot be added with a protocol header (http).

{

"insecure-registries": ["harbor.xxx.cn:30002"]

}

2. Configure the storage driver

OverlayFS is a new generation of federated file system, similar to AUFS, but faster and easier to implement. Docker provides two storage drivers for OverlayFS: the old version of overlay and the new version of overlay2 (more stable).

Prerequisites:

Overlay2: Linux kernel version 4.0 or later, or RHEL or CentOS with kernel version 3.10.0-514 +.

Overlay: host Linux kernel version 3.18 +

Supported disk file system

Ext4 (RHEL 7.1 only)

Xfs (RHEL7.2 and later), d_type=true needs to be enabled.

{

"storage-driver": "overlay2"

"storage-opts": ["overlay2.override_kernel_check=true"]

}

3. Configure the log driver

The container generates a large number of log files at run time, which can easily fill up disk space. Limit the file size and number of files by configuring the log driver. > limit a single log file to 50m, and generate a maximum of 3 log files

{

"log-driver": "json-file"

"log-opts": {

"max-size": "50m"

"max-file": "3"

}

}

The final configuration file is as follows:

{"insecure-registries": ["harbor.xxx.cn:30002"], "storage-driver": "overlay2", "storage-opts": ["overlay2.override_kernel_check=true"], "log-driver": "json-file", "log-opts": {"max-size": "50m", "max-file": "3"} 3, install RKE

Rancher Kubernetes Engine (RKE) is a lightweight Kubernetes installer that supports Kubernetes installation on bare metal and virtualized servers. RKE solves a common problem in the Kubernettes community, such as installation complexity. RKE supports a variety of platforms, such as MacOS,linux,windows.

Install rke on master1 here:

1. Download binaries

Https://www.rancher.cn/docs/rancher/v2.x/cn/install-prepare/download/rke/

2. Run the command test:

Chmod + x rke_linux-amd64./rke_linux-amd64-- version4, install kubectl

Kubectl is a CLI command-line tool that runs commands for Kubernetes clusters. It is needed for many maintenance and management in Rancher 2.x.

Install kubectl on master1 here:

1. Download binaries

Https://www.rancher.cn/docs/rancher/v2.x/cn/install-prepare/download/kubernetes/

2. Make sure that the kubectl binaries are executable.

Chmod + x. / kubectl

3. Move the kubectl binaries to the PATH path.

Sudo mv. / kubectl / usr/local/bin/kubectl

4. Configure kubectl

When you create a Kubernetes cluster using RKE, RKE creates a profile kube_config_rancher-cluster.yml that contains authentication information in the local directory to connect to the new cluster using tools such as kubectl or helm.

You can copy this file to $HOME/.kube/config

Cp kube_config_rancher-cluster.yml ~ / .kube/configexport KUBECONFIG=$ (pwd) / kube_config_rancher-cluster.yml

4. Configure shell completion of kubectl

On CentOS Linux, you may need to install bash-completion packages that are not installed by default.

Yum install bash-completion-y

Run source > ~ / .bashrc II, install 1, configure load balancer

Use Tencent * Cloud load balancer to import the domain name ssl certificate (the domain name here must be configured to access the same domain name. I use the free one-year ssl certificate ♪ (^ ∇ ^). Bind port 10443 to port 80 of master1,master2,master3.

Note: the corresponding security rules need to be activated, that is, the ip of load balancer can access ports 80 of master1, 2 and 3.

2. RKE installation K8S (1) create rke installation file on master1 rancher-cluster.yml#vim rancher-cluster.ymlnodes:-address: xxx.xxx.xxx.xxx internal_address: 172.27.100.101 user: rancher role: [controlplane,worker,etcd] hostname_override: master1-address: xxx.xxx.xxx.xxx internal_address: 172.27.100.102 user: rancher role: [controlplane,worker Etcd] hostname_override: master2-address: xxx.xxx.xxx.xxx internal_address: 172.27.100.103 user: rancher role: [controlplane,worker,etcd] hostname_override: master3services: etcd: backup_config: enabled: true interval_hours: 6 retention: 60

Note:

Address public domain name or IP address

User users who can run the docker command

List of Kubernetes roles assigned by role to nodes

Private domain name or IP address of internal_address internal cluster communication

Etcd backup mechanism is enabled to back up every 6 hours and save data for 60 days.

(2) create K8S cluster and test

# rke up-- config. / rancher-cluster.yml

When you are finished, a file kube_config_rancher-cluster.yml is created. This file contains the credentials for kubectl and helm to access K8S. You can copy this file to $HOME/.kube/config, or if you are using multiple Kubernetes clusters, set the KUBECONFIG environment variable to the kube_config_rancher-cluster.yml file path.

Note: save kube_config_rancher-cluster.yml and rancher-cluster.yml, later maintenance and upgrade need to be used!

Export KUBECONFIG=$ (pwd) / kube_config_rancher-cluster.ymlcp kube_config_rancher-cluster.yml / root/.kube/config/

Use kubectl get nodes to test:

3. Install and configure Helm

Helm is the preferred package management tool for Kubernetes. Helmcharts provides template syntax for Kubernetes YAML manifest documents. With Helm, you can create a configurable deployment, not just using static files. Helm has two parts: the Helm client (helm) and the Helm server (Tiller).

(1) configure Helm client access permissions

On master1, run under the same directory as the kube_config_rancher-cluster.yml file generated in the previous step:

Kubectl-- kubeconfig=kube_configxxx.yml-n kube-system create serviceaccount tillerkubectl-- kubeconfig=kube_configxxx.yml create clusterrolebinding tiller\-- clusterrole cluster-admin-- serviceaccount=kube-system:tiller

Note: create a ServiceAccount; in the kube-system namespace create a ClusterRoleBinding to grant the tiller account access to the cluster; helm initializes the tiller service

(2) install Helm client

1. Download helm:

Https://www.rancher.cn/docs/rancher/v2.x/cn/install-prepare/download/helm/

2. Decompress and configure:

Tar-zxvf helm-v2.x.x-linux-amd64.tgz

Helm finds the binaries in the extracted directory and moves them to the desired location

Mv linux-amd64/helm / usr/local/bin/helm & & chmod + x / usr/local/bin/helm

(3) install Helm server (Tiller)

On master1, run under the same directory as the kube_config_rancher-cluster.yml file generated in the previous step:

Kubeconfig=xxx.ymlhelm_version= `helm version | grep Client | awk-F ""\ "{print $2} '`helm init-- kubeconfig=$kubeconfig\-- service-account tiller--skip-refresh\-- tiller-image registry.cn-shanghai.aliyuncs.com/rancher/tiller:$helm_version

Note:

1. RKE enables RBAC by default, so you need to specify ServiceAccount when installing tiller.

2. In the default configuration, helm init will go to Google Image Warehouse to pull gcr.io/kubernetes-helm/tiller images, and install and configure Tiller; on Kubernetes clusters. Since gcr.io, storage.googleapis.com and other domain names may not be accessible in China, you can specify private image repository images through-- tiller-image.

3. In the default configuration, helm init will use https://kubernetes-charts.storage.googleapis.com as the default stable repository address and update the relevant index files. The storage.googleapis.com address may not be accessible in China. You can specify the chart domestic accelerated image address through-- stable-repo-url.

4. If you are installing Tiller offline, if you do not have an internal chart repository, you can prevent Tiller from updating the index by adding the-- skip-refresh parameter.

4. Helm installation Rancher (1) add Chart warehouse address helm repo add rancher-stable\ https://releases.rancher.com/server-charts/stable (2) configure SSL and install Rancher Server

SSL certificate uses an one-year free certificate provided by Tencent Cloud. Self-signed certificate can be found on the official website.

Export KUBECONFIG=/home/rancher/kube_config_rancher-cluster.ymlhelm-kubeconfig=$KUBECONFIG install rancher-stable/rancher-name rancher--namespace cattle-system-set hostname=paas.yunjingtech.cn-set tls=external

After the installation is successful, enter the previously configured domain name ~

Note:

Check the system space after login. If cattle-cluster-agent Pod and cattle-node-agent do not work properly, you need to add a CVM alias for them:

Export kubeconfig=xxx/xxx/xx.kubeconfig.ymlkubectl-- kubeconfig=$kubeconfig-n cattle-system\ patch deployments rancher-- patch'{"spec": {"template": {"spec": {"hostAliases": [{"hostnames": ["xxx.cnrancher.com"] "ip": "xxxxxx"} 'export kubeconfig=xxx/xxx/xx.kubeconfig.ymlkubectl-- kubeconfig=$kubeconfig-n cattle-system\ patch deployments cattle-cluster-agent-- patch' {"spec": {"template": {"spec": {"hostAliases": [{"hostnames": ["demo.cnrancher.com"] "ip": "xxxxxx"} 'export kubeconfig=xxx/xxx/xx.kubeconfig.ymlkubectl-- kubeconfig=$kubeconfig-n cattle-system\ patch daemonsets cattle-node-agent-- patch' {"spec": {"template": {"spec": { "hostAliases": [{"hostnames": ["xxx.rancher.com"] "ip": "xxxxxx"}]}'

The biggest problem encountered during rancher installation is that security groups are not configured in some places! Fortunately, it has been solved in the end, and we will continue to share the problems encountered in the process of using CICD on the rancher platform, deploying persistent storage Ceph, deploying Harbor, Gitlab, Nexus3, etc., and how to deal with them ~ (^ U ^) YO.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.