Shulou(Shulou.com)05/31 Report--

This article mainly explains "how to configure Flannel". Friends who are interested might as well take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn how to configure Flannel.

The configuration of Flannel is configured through the command line

The command line configuration and instructions for the latest version of Flannel v0.10.0 are as follows:

Usage: / opt/bin/flanneld [OPTION]... -etcd-cafile string SSL Certificate Authority file used to secure etcd communication-etcd-certfile string SSL certification file used to secure etcd communication-etcd-endpoints string a comma-delimited list of etcd endpoints (default "http://127.0.0.1:4001, Http://127.0.0.1:2379")-etcd-keyfile string SSL keyfile used to secure etcd communication-etcd-password string password for BasicAuth to etcd- etcd-prefix string etcd prefix (default "/ coreos.com/network")-etcd-username string username for BasicAuth to etcd- healthz-ip string the IP address for healthz server to listen (default "0.0.0.0" -healthz-port int the port for healthz server to listen (0 to disable)-iface value interface to use (IP or name) for inter-host communication. Can be specified multiple times to check each option in order. Returns the first match found. -iface-regex value regex expression to match the first interface to use (IP or name) for inter-host communication. Can be specified multiple times to check each regex in order. Returns the first match found. Regexes are checked after specific interfaces specified by the iface option have already been checked. -ip-masq setup IP masquerade rule for traffic destined outside of overlay network-kube-api-url string Kubernetes API server URL. Does not need to be specified if flannel is running in a pod. -kube-subnet-mgr contact the Kubernetes API for subnet assignment instead of etcd. -kubeconfig-file string kubeconfig file location. Does not need to be specified if flannel is running in a pod. -log_backtrace_at value when logging hits line file:N, emit a stack trace-public-ip string IP accessible by other nodes for inter-host communication-subnet-file string filename where env variables (subnet, MTU,...) Will be written to (default "/ run/flannel/subnet.env")-subnet-lease-renew-margin int subnet lease renewal margin, in minutes, ranging from 1 to 1439 (default 60)-v value log level for V logs-version print version and exit-vmodule value comma-separated list of pattern=N settings for file-filtered logging

It needs to be explained as follows:

We get the configuration by reading the corresponding ConfigMap from the Kubernetes APIServer through the-kube-subnet-mgr configuration Flannel. -kubeconfig-file,-kube-api-url we also do not configure, because we use DaemonSet to deploy Flannel through Pod, so Flannel and Kubernetes APIServer authenticate communication through ServiceAccount.

Another way is to read the Flannel configuration directly from etcd, and you need to configure the corresponding Flag starting with-etcd.

-subnet- file defaults to / run/flannel/subnet.env, which generally does not need to be changed. Flannel injects the environment variables corresponding to the native subnet information into the file, where Flannel actually obtains the subnet information, such as:

FLANNEL_NETWORK=10.244.0.0/16FLANNEL_SUBNET=10.244.26.1/24FLANNEL_MTU=1500FLANNEL_IPMASQ=true

-subnet-lease-renew-margin indicates that the etcd lease can be renewed automatically before the lease expires. The default is 1h. Because the ttl time is 24 hours, this configuration is naturally not allowed to exceed 24 hours, that is, [1, 1439] min.

Configure through environment variabl

The above command line configuration items can be changed to uppercase, underscore to dash, and FLANNELD_ prefix to the corresponding environment variable to set.

For example, the environment variable for etcd-endpoints= http://10.0.0.2:2379 is FLANNELD_ETCD_ENDPOINTS= http://10.0.0.2:2379.

Deploy Flannel

There is no dispute about deploying Flannel through Kubernetes DaemonSet. At the same time, create the corresponding ClusterRole,ClusterRoleBinding,ServiceAccount,ConfigMap. The complete Yaml description file can be referenced as follows:

-kind: ClusterRoleapiVersion: rbac.authorization.k8s.io/v1beta1metadata: name: flannelrules:-apiGroups:-"" resources:-pods verbs:-get-apiGroups:-"" resources:-nodes verbs:-list-watch-apiGroups:-"" resources:-nodes/status verbs:-patch---kind: ClusterRoleBindingapiVersion: Rbac.authorization.k8s.io/v1beta1metadata: name: flannelroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannelsubjects:- kind: ServiceAccount name: flannel namespace: v1kind: ServiceAccountmetadata: name: flannel namespace: kube-system---apiVersion: v1kind: ConfigMapmetadata: name: kube-flannel-cfg namespace: kube-system labels: tier: node k8s-app: flanneldata: cni-conf.json: {"name": "cbr0" "plugins": [{"type": "flannel", "delegate": {"hairpinMode": true, "isDefaultGateway": true}}]} net-conf.json: | {"Network": "10.244.0.0amp 16" "Backend": {"Type": "host-gw"}-apiVersion: extensions/v1beta1kind: DaemonSetmetadata: name: kube-flannel namespace: kube-system labels: tier: node k8s-app: flannelspec: template: metadata: labels: tier: node k8s-app: flannelspec: imagePullSecrets:-name: harborsecret serviceAccountName: flannel containers: -name: kube-flannel image: registry.vivo.xyz:4443/coreos/flannel:v0.10.0-amd64 command: ["/ opt/bin/flanneld" "- ip-masq" "- kube-subnet-mgr"] securityContext: privileged: true env:-name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name-name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace-name: POD_IP valueFrom: FieldRef: fieldPath: status.podIP volumeMounts:-name: run mountPath: / run-name: cni mountPath: / etc/cni/net.d-name: flannel-cfg mountPath: / etc/kube-flannel/-name: install-cni image: registry.vivo.xyz:4443/coreos/flannel-cni:v0.3.0 Command: ["/ install-cni.sh"] # command: ["sleep" "10000"] env: # The CNI network config to install on each node. -name: CNI_NETWORK_CONFIG valueFrom: configMapKeyRef: name: kube-flannel-cfg key: cni-conf.json volumeMounts: #-name: cni # mountPath: / etc/cni/net.d-name: cni mountPath: / host/etc/cni/net.d-name: host-cni-bin MountPath: / host/opt/cni/bin/ hostNetwork: true tolerations:-key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule volumes:-name: run hostPath: path: / run #-name: cni # hostPath: # path: / etc/kubernetes/cni/net.d -name: cni hostPath: path: / etc/cni/net.d-name: flannel-cfg configMap: name: kube-flannel-cfg-name: host-cni-bin hostPath: path: / etc/cni/net.d updateStrategy: rollingUpdate: maxUnavailable: 1 type: RollingUpdate working principle

It's easy to confuse a few things. When we usually talk about Flannel (coreos/flannel), we actually mean flanneld. Everyone knows that Kubernetes connects network plug-ins through the CNI standard, but when you look at the code of Flannel (coreos/flannel), you don't find that it implements the interface of CNI. If you have played with other CNI plug-ins, you will know that there is also a binary file for kubele to call, and will call the back-end network plug-in. What is this binary file for Flannel (coreos/flannel)? Where is git repo?

This binary file corresponds to the host's / etc/cni/net.d/flannel, its code address is https://github.com/containernetworking/plugins, the most hateful of which is called flannel, why not name it flannelk8s like the contivk8s corresponding to contiv netplugin.

There is also a container called install-cni in the above Flannel Pod, and its corresponding script is in https://github.com/coreos/flannel-cni.

/ opt/bin/flanneld-- > https://github.com/coreos/flannel

/ etc/cni/net.d/flannel-- > https://github.com/containernetworking/plugins

/ install-cni.sh-- > https://github.com/coreos/flannel-cni

Kube-flannel container

Running in the kube-flannel container is our protagonist flanneld. We need to pay attention to the directories / files in the container:

/ etc/kube-flannel/cni-conf.json

/ etc/kube-flannel/net-conf.json

/ run/flannel/subnet.env

/ opt/bin/flanneld

Here is the corresponding content of my environment:

/ run/flannel # ls / etc/kube-flannel/cni-conf.json net-conf.json/run/flannel # cat / etc/kube-flannel/cni-conf.json {"name": "cbr0", "plugins": [{"type": "flannel", "delegate": {"hairpinMode": true "isDefaultGateway": true}}]} / run/flannel # cat / etc/kube-flannel/net-conf.json {"Network": "10.244.0.0and16" "Backend": {"Type": "host-gw"} / run/flannel # cat / run/flannel/subnet.env FLANNEL_NETWORK=10.244.0.0/16FLANNEL_SUBNET=10.244.26.1/24FLANNEL_MTU=1500FLANNEL_IPMASQ=true/run/flannel # ls / opt/bin/flanneld mk-docker-opts.sh/run/flannel # cat / opt/bin/mk-docker-opts.sh #! / bin/shusage () {echo " $0 [- f FLANNEL-ENV-FILE] [- d DOCKER-ENV-FILE] [- I] [- c] [- m] [- k COMBINED-KEY] Generate Docker daemon options based on flannel env fileOPTIONS:-f Path to flannel env file. Defaults to / run/flannel/subnet.env-d Path to Docker env file to write to. Defaults to / run/docker_opts.env-i Output each Docker option as individual var. E.g. DOCKER_OPT_MTU=1500-c Output combined Docker options into DOCKER_OPTS var-k Set the combined options key to this value (default DOCKER_OPTS=)-m Do not output-- ip-masq (useful for older Docker version) "> & 2 exit 1} flannel_env=" / run/flannel/subnet.env "docker_env=" / run/docker_opts.env "combined_opts_key=" DOCKER_OPTS "indiv_opts=falsecombined _ opts=falseipmasq=truewhile getopts "f:d:icmk:?h" opt Do case $opt in f) flannel_env=$OPTARG;; d) docker_env=$OPTARG;; I) indiv_opts=true C) combined_opts=true;; m) ipmasq=false;; k) combined_opts_key=$OPTARG [\ h]) usage;; esacdoneif [$indiv_opts= false] & & [$combined_opts= false]; then indiv_opts=true combined_opts=truefiif [- f "$flannel_env"]; then. $flannel_envfiif [- n "$FLANNEL_SUBNET"]; then DOCKER_OPT_BIP= "--bip=$FLANNEL_SUBNET" fiif [- n "$FLANNEL_MTU"]; then DOCKER_OPT_MTU= "- mtu=$FLANNEL_MTU" fiif [- n "$FLANNEL_IPMASQ"] & & [$ipmasq = true]; then if ["$FLANNEL_IPMASQ" = true] Then DOCKER_OPT_IPMASQ= "- ip-masq=false" elif ["$FLANNEL_IPMASQ" = false]; then DOCKER_OPT_IPMASQ= "--ip-masq=true" else echo "Invalid value of FLANNEL_IPMASQ: $FLANNEL_IPMASQ" > & 2 exit 1 fifieval docker_opts= "\ ${combined_opts_key}" if ["$docker_opts"] Then docker_opts= "$docker_opts"; fiecho-n "" > $docker_envfor opt in $(set | grep "DOCKER_OPT_"); do OPT_NAME=$ (echo $opt | awk-F "="'{print $1;}'); OPT_VALUE=$ (eval echo "\ $$OPT_NAME"); if ["$indiv_opts" = true]; then echo "$OPT_NAME=\" $OPT_VALUE\ "> $docker_env Fi docker_opts= "$docker_opts $OPT_VALUE"; doneif ["$combined_opts" = true]; then echo "${combined_opts_key} =\" ${docker_opts}\ "> $docker_envfiinstall-cni container

As the name implies, the install-cni container is responsible for installing the cni plug-in. Copy the binary files such as flannel in the image to the / etc/cni/net.d of the host. Note that this directory should match the cni configuration item corresponding to kubelet. If you have not changed the default configuration of kubelet, then kubelet defaults to this cni directory. We need to pay attention to the directories / files in the install-cni container:

/ host/etc/cni/net.d/

/ host/opt/cni/bin/

/ host/etc/cni/net.d/10-flannel.conflist

Here is the corresponding content of my environment:

/ host/etc/cni/net.d # pwd/host/etc/cni/net.d/host/etc/cni/net.d # ls10-flannel.conflist dhcp ipvlan noop tuningbridge flannel loopback portmap vlancnitool host-local macvlan ptp/host/etc / cni/net.d # cd / host/opt/cni/bin//host/opt/cni/bin # ls10-flannel.conflist dhcp ipvlan noop tuningbridge flannel loopback portmap vlancnitool host-local macvlan ptp/opt/cni/bin # lsbridge dhcp Host-local loopback noop ptp vlancnitool flannel ipvlan macvlan portmap tuning/opt/cni/bin # cat / host/etc/cni/net.d/10-flannel.conflist {"name": "cbr0" "plugins": [{"type": "flannel", "delegate": {"hairpinMode": true, "isDefaultGateway": true}]} Flannel working schematic diagram

Draw a picture, it should be very clear. Note that the colored part is the corresponding information of Volume, which can be paid more attention to.

The process of creating a container network is: kubelet-- > flannel-- > flanneld. If Pod is created concurrently on the host machine, you will see multiple flannel processes in the background, but normally it ends in a few seconds, while flanneld is a resident process.

Flannel host-gw Data Flow

Openshift also uses the Flannel host-gw container network solution by default, and its official website also clearly shows the data flow diagram of host-gw:

The corresponding ip routes in Node 1:

Default via 192.168.0.100 dev eth0 proto static metric 10010.1.15.0/24 dev docker0 proto kernel scope link src 10.1.15.110.1.20.0/24 via 192.168.0.200 dev eth0

The corresponding ip routes in Node 2:

Default via 192.168.0.200 dev eth0 proto static metric 10010.1.20.0/24 dev docker0 proto kernel scope link src 10.1.20.110.1.15.0/24 via 192.168.0.100 dev eth0

Considerations for using Flannel in Kubernetes Cluster

In my cluster, I use kube-subnet-mgr to manage subnet, not directly through etcd v2.

When flanneld starts, you need to have PodCIDR configured on the Node. You can check whether the .spec.PodCIDR field has a value through the get node information.

There are two ways to configure CIDR for Node:

Manually configure kubelet's-- pod-cidr on each Node

Configure kube-controller-manager-- allocate-node-cidrs=true-- cluster-cidr=xx.xx.xx.xx/yy, and CIDR Controller automatically configures PodCIDR for each node.

In addition, you will find that each Node is marked with a lot of Annotation with the beginning of flannel, and these Annotation will be updated every time flanneld starts RegisterNetwork. These Annotation are mainly used for Node Lease.

Flannel.alpha.coreos.com/backend-data: "null"

Flannel.alpha.coreos.com/backend-type: host-gw

Flannel.alpha.coreos.com/kube-subnet-manager: "true"

Flannel.alpha.coreos.com/public-ip: xx.xx.xx.xx

Flannel.alpha.coreos.com/public-ip-overwrite:yy.yy.yy.yy (ps:optional)

Here is the information about a node in my environment:

# kubectl get no 10.21.36.79-o yamlapiVersion: v1kind: Nodemetadata: annotations: flannel.alpha.coreos.com/backend-data: "null" flannel.alpha.coreos.com/backend-type: host-gw flannel.alpha.coreos.com/kube-subnet-manager: "true" flannel.alpha.coreos.com/public-ip: 10.21.36.79 node.alpha.kubernetes.io/ttl: "0" volumes.kubernetes.io/ Controller-managed-attach-detach: "true" creationTimestamp: 2018-02-09T07:18:06Z labels: beta.kubernetes.io/arch: amd64 beta.kubernetes.io/os: linux kubernetes.io/hostname: 10.21.36.79 name: 10.21.36.79 resourceVersion: "45074326" selfLink: / api/v1/nodes/10.21.36.79 uid: 5f91765e-0d69-11e8-88cb-f403434bff24spec: externalID: 10.21.36.79 podCIDR: 10. 244.29.0/24status: addresses:-address: 10.21.36.79 type: InternalIP-address: 10.21.36.79 type: Hostname allocatable: alpha.kubernetes.io/nvidia-gpu: "0" cpu: "34" memory: 362301176Ki pods: "200" capacity: alpha.kubernetes.io/nvidia-gpu: "0" cpu: "40" memory: 395958008Ki pods: "200" conditions LastHeartbeatTime: 2018-02-27T14:07:30Z lastTransitionTime: 2018-02-13T13:05:57Z message: kubelet has sufficient disk space available reason: KubeletHasSufficientDisk status: "False" type: OutOfDisk-lastHeartbeatTime: 2018-02-27T14:07:30Z lastTransitionTime: 2018-02-13T13:05:57Z message: kubelet has sufficient memory available reason: KubeletHasSufficientMemory status: "False" type: MemoryPressure-lastHeartbeatTime: 2018-02-27T14:07:30Z lastTransitionTime: 2018-02- 13T13:05:57Z message: kubelet has no disk pressure reason: KubeletHasNoDiskPressure status: "False" type: DiskPressure-lastHeartbeatTime: 2018-02-27T14:07:30Z lastTransitionTime: 2018-02-13T13:05:57Z message: kubelet is posting ready status reason: KubeletReady status: "True" type: Ready daemonEndpoints: kubeletEndpoint: Port: 10250 images:-names:-registry.vivo.xyz:4443/bigdata_release/tensorflow1.5.0@sha256:6d61595c8e85d3724ec42298f8f97cdc782c5d83dd8f651c2eb037c25f525071 -registry.vivo.xyz:4443/bigdata_release/tensorflow1.5.0:v2.0 sizeBytes: 3217838862-names:-registry.vivo.xyz:4443/bigdata_release/tensorflow1.3.0@sha256:d14b7776578e3e844bab203b17ae504a0696038c7106469504440841ce17e85f-registry.vivo.xyz:4443/bigdata_release/tensorflow1.3.0:v1.9 sizeBytes: 2504726638-names:-registry.vivo.xyz:4443/coreos/flannel-cni@sha256:dc5b5b370700645efcacb1984ae1e48ec9e297acbb536251689a239f13d08850-registry.vivo.xyz:4443/coreos / flannel-cni:v0.3.0 sizeBytes: 49786179-names:-registry.vivo.xyz:4443/coreos/flannel@sha256:2a1361c414acc80e00514bc7abdbe0cd3dc9b65a181e5ac7393363bcc8621f39-registry.vivo.xyz:4443/coreos/flannel:v0.10.0-amd64 sizeBytes: 44577768-names:-registry.vivo.xyz:4443/google_containers/pause-amd64@sha256:3b3a29e3c90ae7762bdf587d19302e62485b6bef46e114b741f7d75dba023bd3-registry.vivo.xyz:4443/google_containers/pause-amd64:3.0 sizeBytes: 746888 nodeInfo: architecture: Amd64 bootID: bc7a36a4-2d9b-4caa-b852-445a5fb1b0b9 containerRuntimeVersion: docker://1.12.6 kernelVersion: 3.10.0-514.el7.x86_64 kubeProxyVersion: v1.7.4+793658f2d7ca7 kubeletVersion: v1.7.4+793658f2d7ca7 machineID: edaf7dacea45404b9b3cfe053181d317 operatingSystem: CentOS Linux 7 (Core) systemUUID: 30393137-3136-4336-5537-3335444C4C30 I believe that you have a deeper understanding of "how to configure Flannel", you might as well come to the actual operation! Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.