Shulou(Shulou.com)06/01 Report--

This article mainly introduces the example analysis of Kubernetes architecture, the article is very detailed, has a certain reference value, interested friends must read it!

First of all, why use Kubernetes? To use a tool, you have to sort out the goal of using the tool. We don't use the tool for the sake of the tool.

Kubernetes's goal is best illustrated by a picture that has been quoted by many people:

In a word, the goal of Kubernetes is to allow you to manage your services like livestock, not like pets, while improving resource utilization and allowing programmers to focus on application development itself, leaving high-availability matters to Kubernetes. This diagram was originally proposed by openstack, but a pure IaaS-tier solution could not achieve this goal, so Kubernetes came into being.

Kubernetes is the same as Borg, which is basically an improved open source version of Borg, quoting what Google Borg said in his paper:

It (1) hides the details of resource management and failure handling so its users can focus on application development instead; (2) operates with very high reliability and availability, and supports applica- tions that do the same; and (3) lets us run workloads across tens of thousands of machines effectively

How can we verify that this goal has been achieved?

Turn off a machine to see if your service is normal.

Can reduced application instances be automatically migrated and restored to other nodes

Can the service scale automatically with traffic

Let's explore the architecture improvement of a simple multi-tier application:

Description:

Mysql should be an one-master, multi-slave architecture, which is omitted for simplicity.

Service will also rely on resources such as databases, which are omitted here for simplicity

Arrows represent calls and dependencies

In order to achieve our goal, we need to make improvements:

If Loadbalancer invokes the backend application service node, if the backend application service node dies or migrates additional nodes, the configuration of Loadbalancer will be changed. This obviously fails to achieve the goal, so it is planned to transform Loadbalancer into Smart Loadbalancer. Through the service discovery mechanism, the application instance is automatically registered to a configuration center (etcd/zookeeper) when the application instance is started or terminated, and the Loadbalancer listens for changes in the application configuration to automatically modify its own configuration.

Web applications rely on backend resources, such as Mysql and Memcached, and the ip of the corresponding resources is generally written to the configuration file. The application configuration should be changed when the resource node is changed or added.

Mysql plans to use domain name access instead of ip. In order to avoid the delay of dns changes, it is necessary to set up a private dns on the private network. High availability adopts the MHA scheme, and then automatically modifies the dns with the service discovery mechanism.

The Memcached plan refers to the way of couchbase, through the service discovery mechanism, using SmartClient, the client application listens for changes in the nodes of the configuration center. The difficulty may lie in the modification of Memcached (see couchbase). In addition, it can also be achieved by adding a layer of proxy.

How to deal with the difference between the system and the underlying library on which the application node migrates? How to deal with different deployment methods? What about conflicts such as disk paths, listening ports, etc.? This can standardize the way the application is deployed and run through container technology such as Docker, the dependency of the operating system and the underlying library allows the application to be customized, and the dependence on the disk path and port is dynamically injected through the Docker running parameters without changing the application configuration. Docker custom variables and parameters need to provide a standardized configuration file.

Service migration problem each service needs a master scheduling center to monitor the status of the instance, determine whether to migrate or not, and be responsible for unified scheduling. And there should be an agent on each server node to perform specific operations and monitor the applications on that node. In addition, interfaces and tools are provided to operate.

The problem of network and port conflicts is troublesome and requires the introduction of solutions similar to SDN.

Memory, cpu, disk and other hardware resources, the original habit is to buy the server according to the type of application on the server planning, if the application and hardware decoupling, this approach needs to be eliminated. However, there must be a scheduling mechanism so that applications can be filtered when they migrate.

In summary, through the analysis, it is concluded that the key to achieve the goal is decoupling, the decoupling of application processes and resources (including cpu, memory, disk, network), and the decoupling of service dependencies.

The transformation mechanism above is basically designed on a case-by-case basis, while that of Kubernetes is to provide a comprehensive and universal mechanism.

Then, let's take a look at Kubernetes's solution to the above problem:

Let's start with an official architecture diagram of Kubernetes.

The dispatching center master is mainly composed of four components:

As a configuration center and storage service (Distributed Watchable Storage in the architecture diagram), etcd saves the definition and state of all components, and the interaction between multiple components of Kubernetes is mainly through etcd.

Kubernetes etcd registry's directory structure etcdctl ls / registry/ registry/minions saves node node information / registry/namespaces / registry/pods saves all pods information / registry/ranges / registry/serviceaccounts / registry/services / registry/controllers / registry/events Kubernetes components change events are written to this directory

Kube-apiserver provides interfaces to interact with the outside world and provides security mechanisms. Most interfaces read and write data in etcd directly.

The kube-scheduler scheduler mainly does one thing: listen for pod directory changes in etcd, then assign node through the scheduling algorithm, and finally call the bind interface of apiserver to associate the assigned node with pod (modify the nodeName attribute in the pod node). Scheduler is a plugin in Kubernetes and can be replaced with other implementations (such as mesos). Different algorithms are available. The algorithm interfaces are as follows:

Type ScheduleAlgorithm interface {Schedule (api.Pod, NodeLister) (selectedMachine string, err error)}

Kube-controller-manager undertakes the main functions of master, such as interacting with CloudProvider (IaaS), managing node,pod,replication,service,namespace and so on. The basic mechanism is to listen to the corresponding events under etcd / registry/events and deal with them. The specific logic needs to be analyzed by special articles, which will not be explained in detail here.

The agent on the node has two main components:

Kubelet mainly includes container management, image management, Volume management and so on. At the same time, kubelet is also a rest service, and the command operations related to pod are implemented through the calling interface. For example: check the pod log, execute commands on pod, and so on. The startup and destruction of the pod is still performed by listening for changes to the etcd. However, kubelet does not interact with etcd directly, but through the watch mechanism provided by apiserver, which should be for security reasons. Kubelet provides a plug-in mechanism to support extensions to Volume and Network.

Kube-proxy is mainly used to implement the service mechanism of Kubernetes. Provide some SDN functions as well as intelligent LoadBalancer within the cluster. As we analyzed earlier, one of the difficulties in migrating application instances between multiple server nodes is the problem of network and port conflicts. Kubernetes assigns a clusterIP (virtual ip) to each service. Different service uses different ip, so the ports do not conflict. The virtual ip of Kubernetes is realized through the iptables mechanism. For each port defined by service, kube-proxy listens for a random port and forwards it through iptables nat rules. For example, there is a dns service on Kubernetes, clusterIP:10.254.0.10, port: 53. The application request for 10.254.0.10 pod 53 is forwarded to the random port on which the node's kube-proxy listens, and then forwarded to the corresponding pod. If the pod of the service is not on the current node, it will be forwarded between kube-proxy first. The current version of kube-proxy is implemented through the tcp agent, resulting in a large performance loss (see the stress test comparison below). In version 1.2, it is planned to implement kube-proxy entirely through iptables (https://github.com/kubernetes/kubernetes/issues/3760)

Pods Kubernetes abstracts the concrete example of the application as pod. Each pod starts a google_containers/pause docker container first, and then launches the application of the real docker container. The purpose of this is to make it possible to encapsulate multiple docker containers into a single pod, sharing network addresses.

Replication Controller controls the number of copies of pod, and its high availability depends on it.

Services service is an abstraction of a set of pods. Through the intelligent LoadBalancer mechanism of kube-proxy, the destruction and migration of pods will not affect the function of services and the caller of the upper layer. Kubernetes's abstraction of service can decouple the dependency relationship between underlying services and upper services, and implement an environment variable injection mechanism similar to Docker links (https://github.com/kubernetes/kubernetes/blob/release-1.0/docs/user-guide/services.md#environment-variables), but more flexible. If we cooperate with dns's short domain name resolution mechanism, we can finally achieve complete decoupling.

Tags in Label key-value format are mainly used for filtering. For example, service and backend pod are filtered through label and are weakly associated.

Namespace in Namespace Kubernetes is mainly used to avoid name conflicts in pod,service. The name of the pod,service within the same namespace must be unique.

The command line tool of Kubectl Kubernetes is mainly managed by calling apiserver.

Kube-dns dns is an application on top of Kubernetes. By setting the dns searchDomain of Pod (which is operated when pod is started by kubelet), you can resolve service in the same namespace directly through name resolution (the advantage is that the formal environment for development and testing can share the same set of configuration). It mainly consists of the following components, which are packaged into the same pod.

Etcd skydns dependencies for storing dns data

Skydns open source dns service

Kube2sky listens for internal changes in kube through the interface of apiserver, and then calls the interface of skydns to operate dns

In the concept of Networking Kubernetes, pod can communicate directly with each other (http://kubernetes.io/v1.1/docs/admin/networking.html)), but there is actually no built-in solution, so users need to choose their own solution: Flannel,OpenVSwitch,Weave and so on. We tested with Flannel, which is relatively simple.

The configuration file Kubernetes supports configuration files in yaml and json formats, which are mainly used to define pod,replication controller,service,namespace and so on.

The possible changes brought about by Kubernates

Reduce the complexity of distributed application development and operation and maintenance throughout the current various distributed architectures, hadoop,storm, etc., are master-workers mode, a large part of the functions of the framework in node management, processor scheduling, if based on Kubernetes to achieve similar functions, these basic can be handed over to Kubernetes to complete, the framework is only responsible for the flow of core data and collection logic. Of course, the current pod of Kubernetes does not support batch job as directly as Borg, but according to the relationship between Kubernetes and Borg, it should support (http://blog.kubernetes.io/2015/04/borg-predecessor-to-kubernetes.html)) in the future.

The more complete CI/CD (continuous Integration / continuous delivery) tool CI is a key tool for code-deploy, but at present, due to the inconsistency of the deployment environment, what CI can do is limited, and most of them rely on user-defined scripts. With a standard environment such as Kubernetes, such tools can cover test environment deployment, integration testing, online deployment and other aspects to achieve a standardized delivery workflow.

Distributed storage Kubernetes on Kubernetes officially provides an example of deploying cassandra on Kubernetes. You only need to rewrite a Kubernetes apiserver-based SeedProvider to avoid static configuration of seed ip and enjoy the scale-out capabilities brought by Kubernetes. As in the high-availability example of memcached mentioned earlier, if you implement a memcached smart client based on Kubernetes, you only need to change the client, which is very simple. Personally, I think distributed storage that supports multi-tenancy on Kubernetes will be more popular in the future. As long as the scale-out problem of the storage service is solved, there is generally no big problem with the scale-out of the application. Hypernetes is a Kubernetes version that implements multi-tenancy.

One of the major reasons why SaaS (on-demand) is popular at present is that the deployment and maintenance cost of the original on-premise application is too high. If distributed operating systems such as Kubernetes are widely used, the cost of on-premise will be reduced, and on-premise and managed cloud models will also have an impact on the pattern of SaaS. This is one of the reasons why startups like us focus on Kubernetes.

The impact of IaaS on the components of the current IaaS platform tends to be closed. For example, AWS tries to replace Mysql with Aurora, and Aliyun replaces Redis with KVStore. Users are mainly concerned about the reliability of services, and they may prefer open source solutions when they operate and maintain, but they don't care much if they use the services of cloud vendors. According to this trend, with the popularity of IaaS, it will have an impact on the entire open source ecology. However, with a platform like Kubernetes, IaaS vendors mainly provide the basic environment for Kubernetes to run, and Kubernetes takes over the above applications and services, so it is easy to migrate between IaaS vendors.

Microservices are very popular these days, but this concept is not new. Mainly limited by the complexity of operation and maintenance, there is no popularity. If the operation and maintenance system fails to keep up, and the service is disassembled too carefully, it is easy to find a very old service that is not often updated in the corner of a server. Later, everyone even forgot that the server was lost when it was migrated, and the user complained and found out. With the maturity of tools and platforms such as Docker and Kubernetes, the era of micro-services has come. The micro-service governance framework on Kubernetes can solve the rpc, monitoring and disaster recovery problems of micro-services, which I think we can look forward to.

Some of the problems encountered

Finally, I would like to summarize some of the problems I have encountered.

Wall gcr.io has been wall. if you install it locally in the virtual machine, please climb the wall all the way. If you find a way to download it on the server, then specify the image address in the configuration file.

Concurrent pulling of images leads to image corruption (https://github.com/kubernetes/kubernetes/issues/10623). This is also related to docker. It is recommended to use scripts to pull images on each node before deployment.

Startup sequence of multiple containers in the same pod there is no priority in multiple container definitions of the same pod, and the startup sequence cannot be guaranteed. For example, in kube-dns, etcd starts first, then skydns connects to etcd to create a basic directory, and finally kube2sky starts to synchronize the data already defined in kube to dns. Dns data is not normal if it is not in the right order. If you encounter this kind of problem, just restart the corresponding container in order. At present, this problem needs to be solved by the application itself through the retry mechanism.

Access the external network within the container if the Flannel solution is used, but the public network cannot be accessed in the container (if node can), the iptables is usually broken (https://github.com/coreos/flannel/issues/115)).

The current Kubernetes does not have the concept of application, our application consists of four self-developed service components, as well as some dependencies (mysql,redis,mongodb, etc.), defined to a total of more than 20 yaml. There is still a lot of work to be done to achieve one-click installation or update.

Kubernetes's public network load balancing solution depends on the implementation of IaaS, which is not flexible enough.

For the performance problem of kube-proxy, the simple pressure test results are as follows: 10.254.2.99 pod 80 is the service address, followed by two pod. 11.1.16.15 pod 3000 is one of them. The code is the helloword on the home page of golang's official website.

Ab-c 50-n 100000 http://10.254.2.99:80/ Server Hostname: 10.254.2.99 Server Port: 80 Document Path: / Document Length: 43 bytes Concurrency Level: 50 Time taken for tests: 346.442 seconds Complete requests: 100000 Failed requests: 0 Write errors: 0 Total transferred: 16000000 bytes HTML transferred: 4300000 bytes Requests per second: 288.65 [# / sec] (mean) Time per request: 173.221 [ms] (mean) Time per request: 3.464 [ms] (mean Across all concurrent requests) Transfer rate: 45.10 [Kbytes/sec] received Connection Times (ms) min mean [+ /-sd] median max Connect: 01 39.9 0 7015 Processing: 1173 437.7 22 5026 Waiting: 1172 437.7 22 5026 Total: 3 173 440.4 22 10070 Percentage of the requests served within a certain time (ms) 50% 22 66% 23 75% 24 80% 26% 1022 95% 1023 98% 1031 99% 2024 10070 (longest request) ab-c 50-n 100000 http://11.1.16.15:3000/ Server Hostname: 11.1.16 . 15 Server Port: 3000 Document Path: / Document Length: 43 bytes Concurrency Level: 50 Time taken for tests: 15.992 seconds Complete requests: 100000 Failed requests: 0 Write errors: 0 Total transferred: 16000000 bytes HTML transferred: 4300000 bytes Requests per second: 6253.24 [# / sec] (mean) Time per request: 7.996 [ms] (mean) Time per request: 0.160 [ms] (mean Across all concurrent requests) Transfer rate: 977.07 [Kbytes/sec] received Connection Times (ms) min mean [+ /-sd] median max Connect: 0 002 Processing: 0 8 1.3 7 25 Waiting: 0 8 1 3 7 24 Total: 1 8 1.3 7 25 Percentage of the requests served within a certain time (ms) 50% 7 66 6 8 75 5 8 80% 8 90% 11 95% 11 98% 11 99% 11 100% 25 (longest request)

The above is all the content of the article "sample Analysis of Kubernetes Architecture". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.