Shulou(Shulou.com)05/31 Report--

The editor today takes you to understand what the technical practice of the container management platform based on kubernetes is. The knowledge points in the article are introduced in great detail. Friends who feel helpful can browse the content of the article together with the editor, hoping to help more friends who want to solve this problem to find the answer to the problem. Let's follow the editor to learn more about "what the technical practice of container management platform based on kubernetes is like".

First, the background of container cloud

With the popularity of micro-service architecture, combined with open source micro-service frameworks such as Dubbo and Spring Cloud, many internal business lines of Yixin have gradually shifted from the original single architecture to micro-service architecture. Applications go from stateful to stateless, and specifically store business state data such as sessions, user data and so on into middleware services.

Although the split of micro-service reduces the complexity of each service, the number of service instances shows an explosive growth, which increases the difficulty for operation and maintenance, on the one hand, service deployment and upgrade, and on the other hand, service monitoring fault recovery.

In 2016, container technology, especially Docker, quickly became popular, and the company began to try to put containers into containers to run. Although the service publishing problem has been solved through containers, the operation and maintenance of many containers still make the operation and maintenance staff difficult. Citic is a financial technology company. When introducing open source components, stability and reliability is the most important standard to consider. In early 2017, kubernetes gradually matured and became the container management standard, and was adopted by many domestic and foreign companies. In this context, Yixin draws lessons from open source community and commercial PAAS platform products, and develops a container management platform based on kubernetes.

II. Overall structure

The whole architecture is built around kubernetes and is divided into four tiers. The lowest layer is mainly basic resources, including network, computing, storage, all containers are deployed on physical servers, containers mount commercial NAS storage, and networks are interconnected through vxlan. The core of the middle layer is the resource scheduling layer, which mainly completes multi-cluster management, release and deployment, intelligent scheduling, automatic scaling, etc., this layer is mainly resource management and service orchestration. The left side provides system security, mainly for system security and container image security, while on the right side is a set of code automatic compilation, automatic construction and automatic deployment system; the middleware layer mainly provides commonly used middleware services, Nginx configuration and monitoring alarms, etc.; the top layer is the user access layer, which mainly provides users' operation entry. The overall architecture is shown in the following figure:

III. Nginx self-help Management

Most of the company's services are provided through Nginx reverse proxy. For service isolation and load balancing, there are more than ten sets of Nginx clusters. The versions and configurations of these nginx are different. As a result, the cost of manual operation and maintenance is very high and error-prone, and the IP address of the container is not fixed, which cannot be directly configured to the nginx backend. A self-developed nginx management system is mainly designed to solve the template configuration of nginx, as shown in the following figure:

Nginx-mgr provides HTTP requests, which are responsible for receiving nginx configuration requests and updating them to etcd. Each nginx-agent refreshes the configuration of nginx in batches through watch Etcd. In the actual production environment, Ali open source Tengine is deployed instead of nginx, and there is no distinction because the configuration is basically the same. Each service is configured with a health check to ensure automatic switching in the event of a back-end failure. If there is a virtual airport view that needs to be switched manually, the following figure shows the page of manually switching nginx:

Since many businesses are mixed with virtual machines and containers, if the backend is a container, we use the API of kubernetes to obtain the IP address of the container for dynamic refresh.

IV. Multi-cluster management

Although kubernetes itself uses a highly available deployment architecture to avoid a single point of failure, this is far from enough. On the one hand, because a single kubernetes cluster is deployed in a computer room, if a failure occurs at the computer room level, it will lead to service interruption; on the other hand, due to the failure of a single kubernetes cluster itself, such as the network configuration error of the cluster, the normal use of the business will be affected. In Yixin, kubernetes is deployed in multiple computer rooms, and the computer rooms are interconnected through dedicated lines. Then the management of multiple clusters will become the main difficulty: the first is how to allocate resources. When users choose multi-cluster deployment, the system determines the number of containers allocated to each cluster according to the resource consumption of each cluster, and ensures that each cluster has at least one container. When the cluster automatically scales, containers are also created and recycled according to this ratio. The second is failure migration. For example, the cluster controller in the figure is mainly designed to solve the automatic scaling of multi-clusters and container migration in the event of cluster failures. The controller regularly detects multiple nodes in the cluster. If it fails many times, it will trigger the cluster container migration operation to ensure the reliable operation of the service.

The third is the interconnection of network and storage, because the network across the computer room needs to be interconnected, we use the network scheme of vxlan, and the storage is also interconnected through dedicated lines. The container's image repository uses Harbor, and synchronization policies are set among multiple clusters, and each cluster sets its own domain name resolution, which is resolved to different image repositories.

V. DNS parsing

As business people still have doubts about container technology, most applications are mixed deployment of virtual machines and containers. It is common for containers to access virtual machines through domain names and virtual machines to access containers through domain names. In order to manage domain names uniformly, we use bind instead of kube-dns (coreDns) provided by kubernetes to provide domain name resolution. Point the domain name of the container to the company's DNS server through the Default DNS policy supported by kubernetes, and configure the dynamic addition of API for domain name management.

VI. Network scheme

There are many kinds of CNI network schemes of kubernetes, which are mainly divided into two-layer, three-layer and overlay schemes. On the one hand, the BGP protocol is not allowed in the computer room, and the interconnection of hosts across the computer room is required, so we adopt flannel's vxlan solution. In order to achieve interworking across the computer room, the flannel of the two clusters is connected to the same etcd cluster to ensure the consistency of the network configuration. There are many problems in the old version of Flannel, including too many routes, invalid ARP cache and so on. It is recommended to modify it to the form of network segment routing, and set the ARP rule to be valid forever to avoid cluster network paralysis caused by etcd and other failures.

The use of Flannel also needs to pay attention to some configuration optimization. By default, the lease of Etcd is applied for every day. If the application fails, the etcd IP address range will be deleted. To avoid network segment changes, you can set the ttl of the etcd data node to 0 (never expire). By default, Docker will masq all packets leaving the host, so that the IP address of the source container cannot be obtained in the flannel. Add an exception by setting Ipmasq to exclude the destination address of the flannel network segment packet. Because the flannel uses vxlan, the vxlan offloading that turns on the network card has a high performance improvement. Flannel itself does not have network isolation. In order to implement the network policy of kubernetes, we use canal, which is the plug-in for calico to implement the network policy of kubernetes.

7. CICD

In order to support the Devops process, we tried to perform code compilation using Jenkins in the initial version, but Jenkins had poor support for multi-tenancy. In the second edition, through the Job mechanism of kubernetes, each user's compilation will start a compiled Job. First, the user code will be downloaded, and the corresponding compilation image will be selected according to the compilation language. After compilation, the execution program will be generated, if the jar or war file. The Docker image is made through Dockerfile and pushed to the image repository, and the rolling upgrade process is triggered through the webhook of the image repository.

VIII. Service arrangement

The system designs the logical concept of application. Although kubernetes has the concept of service, it lacks the relationship between services. A complete application usually includes many services, such as front-end, back-end API, middleware and so on. These services call and restrict each other. By defining the concept of application, we can not only control the order of service startup, but also plan to start and stop a group of services.

IX. Log

The log collection of the container uses the watchdog log system developed by the company. Log collection Agent,Agent is deployed on each host through DaemonSet to obtain the container and log path to be collected through Docker API, and logs are collected and sent to the log center, which is based on elasticsearch development and provides multi-dimensional log retrieval and export.

X. Monitoring

The performance monitoring of the resource monitoring of the container itself is done through Cadvisor + Prometheus. The monitoring of the business in the container integrates the open source APM monitoring system uav (https://github.com/uavorg/uavstack) to complete the performance monitoring of the application. Uav's link tracking is based on JavaAgent technology. If you choose to use uav monitoring when deploying an application, the system implants the agent of uav into the image and modifies the startup parameters when building the image.

In addition to the above modules, the system also integrates Harbor to complete the multi-tenant management and image scanning functions of the container image; log audit records users' operations in the management interface, and webshell provides users with access to the web console. In order to support security audit, the background will intercept all users' operation commands in webshell and record them into the database; storage management mainly integrates the company's commercial NAS storage, providing direct data sharing and persistence for containers. The application store mainly provides scenario middleware services for development and testing through kubernetes's operator.

11. Landing practice 11.1 docker is not a virtual machine

At the initial stage of container promotion, business developers are not very familiar with containers and will subconsciously think that containers are virtual machines. In fact, they are not only differences in usage, but also differences in implementation methods and principles. Virtual machines virtualize the hardware environment of the operating system by simulating hardware instructions, while containers provide isolation and restriction of resources on the premise of sharing the kernel. The following figure shows the seven namespace supported by linux in the 4.8kernel.

In other words, nothing else is different, for example, the clock, all containers and operating systems share the same clock, and if you change the time of the operating system, all containers will change. In addition, the container proc file system is not isolated, see is the host information, which brings trouble to many applications, JVM initial heap size of 2G memory limit, and the host is usually 200g memory, JVM is easy to trigger OOM, the solution is usually started based on memory and CPU limits to set JVM, or with the help of lxcfs.

Cgroup's resource limitations currently have weak restrictions on network and disk IO. V1's cgroup only supports direct IO restrictions, but the actual production environment is cached. We are currently also testing cgroup v2's limitations on IO. When the latest CNI already supports network speed limit, combined with tc can achieve this effect very well.

11.2 Kubernetes optimization

Kubernetes comes with a lot of scheduling algorithms, which will pass the scheduling algorithm before starting the container. These algorithms need to filter and sort all the nodes, which greatly increases the deployment time of the container. By deleting some useless scheduling algorithms, the deployment speed is improved. The container adopts an anti-affinity strategy to reduce the impact of physical machine failure on the service.

Although kubernetes enables RBAC, kubernetes token is not recommended to mount it to the business container to improve the security of the system by shutting down ServiceAccountToken.

Docker image storage uses direct-lvm, which has better performance. Separate vg is divided during deployment to avoid affecting the operating system because of Docker problems. Limit the system disk of each container to 10 GB through devicemapper storage to prevent the business container from running out of host disk space. When the container is running, you need to limit the maximum number of processes per container to avoid fork × ×.

Kubernetes core data is recorded in Etcd, so it is necessary to have etcd highly available and scheduled backups. After the kubernetes cluster has more than a hundred nodes, the query speed will slow down, and the speed can be effectively improved through SSD. In addition to kubernetes, the system uses database storage service and

Pay attention to the validity period of certificates. When deploying kubernetes clusters, many of them are self-signed certificates. If not specified, openssl defaults to an one-year validity period. You need to be very careful to update certificates, because the API of the entire kubernetes is built based on certificates, and all associated services need to be modified.

Thank you for your reading, the above is the whole content of "what is the technical practice of the container management platform based on kubernetes". Let's hurry up and operate it. I believe that the editor will certainly bring you better quality articles. Thank you for your support to the website!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.