Shulou(Shulou.com)06/02 Report--

On November 16-17, 2018, CNUTCon Global Operation and maintenance Technology Conference, hosted by InfoQ, was successfully held at Shanghai Everbright Convention and Exhibition Center. Wang Lei, co-founder and CTO of hourly Cloud, was invited to attend the conference and delivered a keynote speech.

The title of Wang Lei's speech is "Operation and maintenance practice under the New Technology Framework of Container PaaS". It explains in detail how to take application as the center in the process of building container PaaS based on Docker + Kubernetes, and how to manage operation and maintenance of services, nodes, clusters, platforms and other aspects through new technologies and tools, so as to improve the automatic operation and maintenance capability of the system. At the same time, combined with the practical experience of building DevOps and micro-service products based on container PaaS, we share how to improve the availability and simplify the cost of operation and maintenance while simplifying DevOps tools and micro-service framework itself.

Wang Lei believes that with the popularity of container technology, container PaaS platform has become an indispensable part of enterprise cloud computing strategy or cloud platform construction; at the same time, container technology has also promoted the gradual standardization and in-depth development of DevOps and micro services, and container PaaS has become an ideal supporting platform for these new concepts, technologies and frameworks. However, during the implementation of the new technical architecture of container PaaS, enterprises and OPS personnel are also faced with the following challenges:

The cost of learning brought about by new technologies and ideas

The complexity brought by the rapid development of technological ecology and how to ensure its stability

Manage the complexity of a high-density, fast-changing runtime environment

How to improve the freedom and innovation ability under the new technology framework

How to carry out cross-center development collaboration-DevOps

Platform support and operation and maintenance under micro-service architecture

Let's first take a look at the main modes of operation and maintenance of the Kubernetes-based container PaaS platform, which are introduced from the perspectives of user service, node, cluster and the operation and maintenance of the platform itself.

The means of user service for operation and maintenance mainly include the following points:

Node failure, automatic migration-set appropriate drive time

Set probe to prevent failure when the service in the container is unresponsive.

Reasonably set the parameters of the probe to ensure uninterrupted service during rolling upgrades.

Using PodDisruptionBudget service availability, PodSecurityPolicy security, defining PriorityClass priorities

Break up hot spots and reschedule through the distribution of services and the use of various resources

Alarm based on service status, number of restarts and other data and duration

Match policies and frequency alarms according to service logs

Configuration version control combined with ConfigMap and gitlab

Give the debugging tool to the user

Service operation audit and unified event management

At the same time, for the support of data middleware, we can deploy operation and maintenance to different middleware clusters by means of CRD and custom operator. Including cluster creation and maintenance, data backup and recovery, storage expansion, etc., can be achieved through different CRD and controller methods, not only to ensure the availability of services, but also to ensure the security of data.

The operation and maintenance of cluster nodes can be considered and applied flexibly from the following points:

Monitoring and alarm of main resource indicators

Node affinity / taint

Mirror and Container gc Policy

Extended Node device Type-ListAndWatch / Allocate

Node maintenance state

Time synchronization

Node failure, custom agent reporting exception

How to deal with the shortage of node resources

-driving strategy

-Node OOM behavior

-Best practices (reserved resources, service QoS, DaemonSet)

As for the operation and maintenance of Kubernetes cluster, it is mainly introduced from the aspects of cluster high availability, federated cluster, resource management, quota management, cluster operation and maintenance tools, cleaning tools and so on. At the same time, on the basis of different underlying IaaS platforms, some capabilities of IaaS can be brought into full play to simplify or improve the operation and maintenance of container PaaS. With the rapid iteration of Kubernetes itself, upgrade has become an aspect that has to be considered. At present, we provide two upgrade paths, in-place or data migration, which are suitable for small version upgrades and large-span version upgrades respectively.

At the same time, for the monitoring, operation and maintenance of the whole platform, we have developed an independent and easy to deploy monitoring platform, which is used to collect the unified state of the development and test image warehouse, production image warehouse, PaaS platform, all kinds of API services, K8s cluster and its core components, each node component, etc., and can monitor the status of related services, and can also trace back the historical status and abnormal situations. Consider the quality of service of each component as a whole.

For the operation and maintenance of the platform, of course, we should also consider the backup and recovery of the data, in order to roll back the data in some scenarios. Our container PaaS also provides regular backup and recovery management of platform and cluster-related data, which can uniformly manage the MySQL data of the platform and the etcd data of each cluster, and also allow access to custom backup sources to achieve unified data management.

Next, let's show you how to build our own DevOps platform based on Kubernetes. First of all, let's talk about what Speed Cloud expects from its own DevOps platform:

Can be more easily integrated with other DevOps or third-party tools

Users have a variety of DevOps requirements and need to have better customization capabilities

Easier to install, operate, expand and scale

Reduce learning costs for customers and within the company

Maintain consistent user experience and data consistency with PaaS platform, and give full play to the existing capabilities of PaaS platform

Help your PaaS and micro-service governance products achieve better DevOps capabilities

The basic architecture of the overall DevOps platform is as follows. Build tasks are managed by custom CRD and operator. Log collection, monitoring alarm, node management, scaling of construction resources, quota management and permission control can all be consistent with the capabilities of the PaaS layer. At the same time, you can make innovations in continuous integration, continuous delivery, continuous deployment and many other resources on the PaaS, such as Pod, Job, CronJob, Volume, ConfigMap, Secret and so on. In the future, the new functions and functional improvements of the PaaS layer can be directly applied to the DevOps platform, which greatly reduces the development and operation and maintenance costs of DevOps.

Next, let's look at some examples of how to implement CI/CD on the DevOps platform:

Realize the construction of docker image

How to manage the production under construction (war package, jar package, etc.)

Realize the integration of tools such as Gitlab/Jenkins/Sonar

Manual audit task

Realize the integration of tools such as Gitlab/Harbor/Jira

Finally, share how to better support the service governance framework on the new technology platform of Container PaaS. It mainly includes how to support the cross-departmental and cross-center collaborative development of micro-services, how to reduce the capability conflicts between the micro-services framework and the PaaS platform, and how to make them better integrate.

In terms of the integration of Spring Cloud and K8s, we can use the open source dependent projects of Spring Cloud and the service discovery, configuration management and other related capabilities of K8s. At the same time, in order to facilitate the management of operation and maintenance, we persist the routing configuration of Zuul using the database and persist the call chain data of Zipkin and the circuit breaker monitoring data of Hystrix respectively, so that the historical data can be traced back at any time. You can also dynamically configure the circuit breaker policy or enable the degradation operation on the micro-service governance platform.

In terms of the integration of Dubbo and K8s, we expand on K8s, customize the dependency package of Dubbo, replace zookeeper, use K8s as the service discovery and registration center, and support communication between dubbo consumer and provider through K8s service or pod ip. Users can choose to use server-side load balancer or Dubbo client-side load balance according to their own needs.

To sum up, we have been committed to creating a container PaaS, DevOps and micro-service governance platform with reliable, simple, automated, integrated extension and collaboration features, hoping to enable users to practice and innovate cloud native applications more quickly and securely. In the future, we will continue to make our efforts in automation, intelligent operation and maintenance and the introduction of ChatOps suitable for container PaaS.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.