Shulou(Shulou.com)05/31 Report--

This article is about how the Bare Metal K8S cluster is used on a large scale in Chick-fil-A. I think it is very practical, so I share it with you. I hope you can get something after reading this article. Let's take a look at it.

Chick-fil-A, an American fast-food chain, uses Kubernetes in the edge computing of its more than 2000 restaurants, which means that there are about 6000 devices running Kubernetes at the same time. One of the biggest challenges related to this is how to deploy and manage so many Kubernetes clusters on the physical machines of the restaurant. This article was written by Chick-fil-A 's technical team to share their experience in the selection of Kubernetes cluster management technology, installation and management of Kubernetes clusters on physical machines.

In most cases, Kubernetes is deployed in the cloud or by a skilled Kubernetes technician on a physical machine (or at least with remote access). But for Chick-fil-A, our Kubernetes deployment was done by installers who focused only on the initial hardware installation. Because of their self-booting nature, they never need to connect directly to computing devices-- instead, they connect Ethernet and power cables, and check the status of the cluster by looking at the application app. The whole replacement process is done by restaurant owners / operators who are not very professional in technology or their teams.

The biggest challenge is that our edge computing deployment is not entirely in the "data center environment".

Cluster Management: an option we have considered

In order to solve the challenges of cluster management, we have done comprehensive technical research and considered the following options:

Kubespray-We initially investigated Ansible-based Kubespray, but we found it quite fragile. When things go well, we get a cluster, but when things don't go well, we create a "brick" that is hard to turn back into a computer. We also found that the process of starting a cluster using Kubespray is very slow, usually spending up to 30 minutes on our hardware stack. We believe that Kubespray can have a longer-term development, but in terms of our research results, we think we need to explore other solutions in other directions.

Openshift-Openshift can create Kubernetes clusters, but we don't like to be too tightly bundled with vendor solutions in critical infrastructure parts and don't want to take the risk of being locked in by technology in the future.

Kops-We are loyal fans of Kops and we use it to deploy our cloud's "dashboard" Kubernetes cluster. Unfortunately, when we use it in our edge computing, Kops is not a viable bare metal solution. We look forward to seeing its development in the future.

Kubeadm-Kubeadm is another good Kubernetes clustering utility. The Kubeadm project looks promising, but we think it is much more complex than some alternatives (especially in terms of flexibility), including.

RKE

As far as our current choice is concerned, RKE is the ultimate winner. RKE is an open source Kubernetes cluster management engine provided by Rancher Labs. Although we don't use Rancher 2.0 to manage our clusters for the time being, we do like to use RKE to initialize and maintain cluster simplicity.

To use RKE, you need to identify a leader node and provide it with a configuration YAML file that contains data about the cluster, mainly the hostnames of the nodes participating in the cluster activity.

If nodes in the cluster are added, deleted, or died, the profile needs to have an accurate description of the current and future nodes. If the configuration cannot be kept up to date, the cluster will fail. Although we believe that the lack of nodes should not cause the cluster initialization / update to fail, this is currently the case.

Installation process

Our installation process in the restaurant is very simple-unpacking the device, plugging it into the power supply and tagged switch ports, and that's it. They automatically start the power supply and implement self-boot and cluster creation. RKE allows non-technical users to perform installation and replacement through an incredibly simple process without understanding Kubernetes or even the overall architecture, which is a great experience, but it does require some more complex boot processes.

Nodes that have not yet been included in the cluster need to coordinate with each other to determine who will be included in the cluster. They also need to select a master node to perform cluster creation through RKE.

Highlander

In order to solve this problem, we developed Highlander. Because we can only have one cluster initiator.

Highlander is part of our basic edge mirror. When each node starts, UDP broadcasts its presence and asks if there is an established leader. It will also start to listen to itself. There is no reply in a few seconds, and it will send another broadcast announcing itself as a leader. Do you have any objection? If there is no objection, the node will soon become the leader of the cluster and respond to all future requests as a leader.

If another node has already declared its role as a leader, the new node will confirm the declaration. Existing leaders will execute "RKE up" to manage the new nodes into the existing cluster.

Nodes communicate regularly to ensure that leaders are still in it. If the old leader is dead, a new leader will be elected through a simple protocol that uses random sleep and leadership statements. Although this is simple, uncomplex, easy to reason and understand, it can work effectively on a scale.

After the leader election, Highlander can also ensure that the cluster is configured correctly. In our environment, this includes:

Switch from KubeDNS to CoreDNS

Create an Istio or other core dashboard node

OAuth identity authentication

Note: each of our nodes has its own identity and a short JWT to access authenticated resources. Highlander provides this identity and provides the token token in the form of a Kubernetes secret key.

Integration process

Although we focus on cluster initialization in this article, we will also introduce the whole process of real-time node initialization in the restaurant.

This is how the Bare Metal K8S cluster is used on a large scale in Chick-fil-A. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.