Shulou(Shulou.com)06/02 Report--

Write at the front

In March 2019, Laguwang began to try to migrate the business of the production environment from UHost to UK8S. By September 2019, most of the business modules of the QA environment had completed the containerization transformation. In the production environment, all the backend management services had been migrated to UK8S, and some business modules had also been containerized. The migration process encountered a lot of problems, but also accumulated some practical experience, and deeply realized the benefits that K8S brought to the enterprise, such as the improvement of resource utilization, the improvement of operation and maintenance efficiency, and the acceleration of business iteration due to environmental consistency.

This paper introduces the container transformation practice based on UK8S from the aspects of business architecture, log collection, monitoring, service exposure / invocation and so on.

Business architecture

As shown in the figure above, the services migrated to UK8S are mainly background management services, but the basic components it relies on are still deployed in UHost. Thanks to the flattened network architecture of UK8S, Pod and VM can be interconnected, so there is no need to change the business architecture in the process of migrating business to UK8S.

All containerized businesses are managed by StatefulSet instead of Deployment. One reason is that the Pod name of StatefulSet is fixed, and it is easy to distribute configuration files through the configuration center, but it is difficult to do stateful publishing if you do configuration distribution based on Deployment. Second, the StatefulSet call chain is very fixed, through the call chain monitoring can quickly find out which Pod has a problem, clearly.

Log collection

Before containerization, the business logs of Laguo.com are written to the local log files of VM. However, with the migration of business to UK8S, because the relationship between Pod (application) and VM is not fixed, once Pod is scheduled to other VM, the application log will be scattered in different VM, which is not convenient for unified collection. Therefore, the containerized part of the application log is selected to be output to a unified log platform system and is not retained locally in VM.

For the log collection solution, Sidecar mode is selected. A filebeat container is built in each business pod. The application container shares the emptyDir log directory with the filebeat container, and the filebeat container is responsible for collecting the master container logs and transferring them to Kafka.

The reason for choosing this scheme is that the logs of the application can still be output to files and do not need to be transformed into stdout and stderr to reduce the burden of business migration to UK8S, and filebeat, as a lightweight collection tool, will not consume too many resources. In addition, SideCar is more flexible than DaemonSet, suitable for large, hybrid clusters, and can achieve tenant isolation, and the logs of different applications can be output to different log systems.

Monitoring scheme

In the choice of monitoring scheme, according to its own situation, pull hook network uses two different schemes for the cluster and business, which are the Prometheus monitoring system built by UCloud and the monitoring system developed by users.

Prometheus is selected at the K8S cluster level. The monitoring at the cluster level is divided into three categories: Node, K8S basic components and K8S resource objects.

For the monitoring of Node, Prometheus provides node-exporter, which can collect data such as CPU, memory, disk IO, disk utilization, network packets, bandwidth and so on.

Kubelet, kube-apiserver, kube-controller-manager and kube-scheduler of K8S basic component classes all provide metrics interface to expose (expose) their own run-time monitoring data, which can be directly pulled by Prometheus deployed in K8S cluster.

In addition, with the combination of cadvisor and kube-state-metrics, the data of CPU, memory, disk IO and network IO of Pod in K8S can be collected directly. It is worth mentioning that the Kube-Prometheus project, which is open source by CoreOS, greatly simplifies the installation, deployment, operation and maintenance of Prometheus. UCloud also provides a branch version adapted to UK8S.

At the business monitoring level, pull and hook follows a previously self-developed monitoring system, which is not only responsible for collecting custom monitoring data, but also responsible for monitoring the health of the overall call chain. Its principle is similar to Prometheus, the application needs to embed SDK, report to the collector through UDP protocol, the collector stores the data directly into OpenTSDB, and then there is a display module (similar to Grafana) to display OpenTSDB data. In addition, in the alarm module, if the monitoring item is found to be higher than the threshold, the display module sends an alarm to the alarm module and generates an event list push to the corresponding person in charge.

Service exposure / invocation

The service exposure of K8S and the invocation between services is a very important issue, especially the architecture of mixed deployment of VM and K8S. To solve this problem, there are many solutions in the community, such as LoadBalancer, Ingress, and so on. Here, pull net directly uses UK8S's own LoadBalancer solution, and exposes services internally through UCloud's private network ULB4, which is easy to operate and has high stability.

The inter-service invocation within the cluster is based on ZK/Eureka service registration and discovery, which is consistent with the previous VM environment and has not been modified.

In addition, there are a large number of basic services such as zk, Kafka, Redis and MySQL. In order to improve the reliability of calls between services, because applications connect these services through domain names, Laguo deploys a set of DNS services based on CoreDNS in the UHost environment. Containerized services and services in VM use this set of DNS services to achieve unified domain name resolution, thus solving the problem of reliability of inter-service calls.

Configuration center

The management and distribution of configuration files, the unified configuration center adopted by Laguo Network, and the secondary development based on Baidu Disconf, so that the connection information such as db can be isolated and issued according to different host names and namespace, which is the reason why K8S resource types use StatefulSet.

The configuration files released by the version are managed uniformly through Git and do not use ConfigMap. On the one hand, it takes into account the impact of excessive ConfigMap on the performance of the cluster, on the other hand, it is consistent with the VM environment.

Continuous delivery

The CI/CD of Laguogu.com operates in four different environments, namely, R & D environment, test environment, pre-release environment (online verification environment), and formal environment. Both pre-release and formal environments run in UCloud's UK8S, and the consistency of the environment is ensured through Namespace isolation.

In addition, Laguogu.com also has a self-developed business publishing system for the VM environment, but this publishing system does not adapt to the container environment. In the K8S environment, Jenkins is used to make the transition, and pipeline is used as the release pipeline. At present, the old business release system is being transformed to be compatible with K8S environment and unify the business release process of the whole company.

Plan for the next step

Pull hook is currently testing HPA (Horizontal Pod Autoscaler) and CA (Cluster Autoscaler), and plans to gradually introduce automatic scaling in the production environment to reduce manual scaling behavior, hoping to reduce IT costs and reduce repetitive work. In addition, with the exception of basic component services, such as MySQL, Kafka, big data cluster, etc., which will continue to use UHost, other services will be gradually migrated to UK8S.

About UK8S

UK8S is a container management service based on Kubernetes. Users can deploy, manage and expand containerized applications on UK8S without paying attention to the operation and maintenance work such as the construction and maintenance of Kubernetes clusters. UK8S is fully compatible with native Kubernetes API, based on UCloud VPC, and integrates cloud products such as ULB, UDisk, EIP, VPC and so on. Welcome to click "https://www.ucloud.cn/site/product/uk8s.html" to learn more about UK8S products."

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.