Shulou(Shulou.com)06/03 Report--

At the beginning of 2016, Docker boomed, and now Kubernetes has attracted thousands of attention. In this era of great need for innovation and speed, the cloud composed of containers, micro-services and DevOps is sweeping the whole IT world. At the recent QCon Global Software Development Conference, Wang Zhihao, a senior research and development engineer for the infrastructure of Getuo application platform, shared his "micro-service practice based on Docker and Kubernetes" based on his years of experience in infrastructure.

Wang Zhihao, senior R & D engineer of GE push application platform infrastructure

I. Micro-service

Micro-service architecture

Microservices divide a single application into multiple tiny services, which are loosely coupled and highly cohesive. Each small service can be developed independently, independent of the specific programming language, and can also use different data storage technologies. Each service can be deployed independently, have its own processes, and communicate with each other through lightweight mechanisms (such as HTTP-based API interface). All services work together to achieve specific business functions.

There are two ways for the client to communicate with the server. The first is that the client communicates directly with each microservice. This architecture has four disadvantages:

(1) multiple service requests with low efficiency

(2) external exposure service interface

(3) the interface protocol cannot be unified.

(4) the client code is complex and the server upgrade is difficult.

The second way is for the API gateway to uniformly proxy each service and provide a unified interface protocol to the outside world. This architecture has three advantages:

(1) encapsulate the details of the service interface to reduce the number of communications

(2) Unified Communications Protocol to reduce client code coupling

(3) Unified authentication, traffic control, and defense

Under this architecture, the gateway may also become a system bottleneck.

Accordingly, these two architectures also bring two ways of service registration discovery, the first is that the client communicates with the service by querying the address of the microservice from the service registry, and the second is to add a unified API gateway to query. The former will increase the complexity of the client, the development cost is high, and the second operation will be more concise, so we chose the second architecture in practice.

With the increase of the number of micro-services, the invocation relationship between services is easy to be coupled, and even circular invocation occurs. The best way to deal with this is to layer the services, that is, to decouple the interdependent services asynchronously through message queuing and other technologies to reduce the dependency between services.

Service layering

The concrete practice of micro-service

1. Technology selection

In practice, our API Gateway uses OpenResty, and OpenResty is based on Nginx and extends support for Lua to build highly concurrent Web services. We realize client-side communication through HTTP interface, the data is basically encapsulated into JSON format, and the communication interface between services is also based on HTTP, and uses message queue for asynchronous decoupling; as for service registration discovery, we choose Lua (to extend the function of API Gateway), Node.js (for developing back-end services), Java (for intensive computing and communication with big data) as the main development language.

two。 Concrete realization process

In practice, we use Lua to develop our own micro-service framework-WebLua, which encapsulates the communication protocols between services and the methods and dependencies of accessing external resources (such as Mysql, Redis, etc.), and provides application slots. We can think of each APP as a functional module, and each APP needs to be plugged into the WebLua to run. WebLua can easily combine modules, either running a micro-service with one APP, or providing services with multiple APP. In this way, developers only need to pay attention to business APP development, which greatly improves the development efficiency. On the right side of the figure is the specific code directory structure. Each APP can be divided into three layers of Action,Page,Data. The Action layer intercepts before and after the request processing, and can do some special processing, such as permission verification before the request; the Page layer mainly parses and verifies the request parameters; the Data layer is responsible for specific business processing, and provides Shell scripts to achieve APP packaging, deployment and installation.

II. API Gateway

An important role in the architecture is the API gateway. Here's an introduction.

As can be seen from the above comparison chart, there is no API Gateway on the left, and many modules, such as Auth,Logging, need to be implemented on their own, resulting in repeated construction of the module, while invading the service, and functional expansion is more difficult; the figure on the right is the architecture diagram after the use of API Gateway, all general-purpose modules are implemented in API Gateway, simple maintenance, one construction, benefit everywhere. In this case, there are higher requirements for API Gateway-its functionality must be easily extensible.

In order to implement such an API gateway, we extend the function of API gateway through plug-ins based on OpenResty and drawing lessons from the plug-in mechanism of Kong and Orange.

As you can see from the API Gateway architecture diagram above, the gateway installs a number of plug-ins, each of which works at one or more stages of the request. The plug-in configuration is updated on Consul and takes effect in real time, and the plug-in rules can be configured flexibly. In the operation, we provide more freedom for plug-in developers, developers can define their own format.

Third, containerization

In the implementation of micro-service, we chose Docker. Here is a detailed introduction to the practice based on Docker.

First of all, the network component chooses Calico, and the service registration discovery and configuration management selects Consul. Consul-Template can monitor changes in Consul configuration and services in real time.

The push image system is based on the CentOS system image, install OpenResty,Nodejs,JDK to get the environment image, and then install the micro-service framework to get the Gorp image. Then install the specific application service on this basis, and get the application service image.

Service registration discovery and configuration update process

In the API gateway, service registration is implemented through Consul-Agent, and configuration updates are implemented through Consul-Template. Consul-Template mainly updates three types of configurations, including: Services: the service addresses of all micro-services of the broker; Products: in a nutshell, the mapping table of requests to micro-services, as shown in the upper left, all requests have a unified specification. You can get Prod from Host and APP from URI, which can dynamically route requests to specific services; Nging-Conf: Nginx configuration of products.

The application service container is registered in the same way as the API gateway. First, the service registers the service with the Consul through the Consul Agent running inside the container, and then monitors and observes the configuration changes on the Consul through Consul-Template, and updates the configuration file. The update of OpenResty or WebNode configuration is to directly overwrite the corresponding configuration file, and then restart the corresponding service.

The above figure is a Docker-based cluster architecture, from which you can see that the Docker cluster consists of three nodes, the whole micro-service is divided into three layers, the top layer is API Gateway, the middle layer is the business layer, and the lowest layer is some basic micro-services common to multi-products.

IV. Kubernetes practice

Although micro-service has many benefits, it also brings many problems, one of which is the complexity of operation and maintenance. In the past, operators only need to face a single application, but now they may be faced with dozens or even hundreds of micro-services. In this case, we need to use Kubernetes to solve the problem. Kubernetes is an open source container orchestration tool from Google that can be used to help manage containers.

At the beginning, when we migrated the container to the Kubernetes cluster, we did not make any change, but used Pod to run all the service systems in the Kubernetes cluster. But with the deeper use of Kubernetes, we have made some changes to microservices.

1. First of all, we use Deployment to deploy the service. Deployment will ensure the survival of a certain number of copies of the service at all times, which improves the stability of the service.

two。 Second, we use Service, which can proxy Pod to achieve load balancing.

3. Kube-DNS can resolve the Service name into a specific ClusterIP, and when the Service is not deleted and rebuilt, its ClusterIP remains unchanged, so the cache parsed by DNS does not have the problem of invalidation. Based on the characteristics of Kube-DNS and Service, we later modified the service registration discovery system.

The figure above shows our current service deployment. Pod is created by Deployment and proxied by Service.

In the process of practice, we also encounter another problem, that is, configuration management.

(1) after microservice, there are many and scattered configuration files.

(2) there are many unnecessary differences between different environments, such as database names.

(3) in many different environments, the same configuration items are exposed to testing and operations.

(4) without version control, rollback is troublesome.

(5) Web UI based on Consul cannot verify illegal input.

In response to these problems, we have made the following adjustments:

(1) unify unnecessary differences between different environments

(2) template the configuration files, expose only the differences, and realize centralized configuration of different configuration files at the same time.

(3) develop the configuration center based on Consul, centrally manage the product configuration, verify the validity of the input, and add version control to facilitate rollback.

Configuration center flow chart

With regard to the log service, we have integrated Fluent-Bit in the application container, configured two input sources, TCP and tail, and two outputs, one is Elasticsearch. All logs are uploaded to ES to display queries through Kibana, and the other is log audit service. Some operation logs that need to be audited are sent to the log audit service for further analysis and processing.

With the increase in the number of microservices, the request link may be extended, and it will be very inconvenient for developers to track problems and troubleshoot performance bottlenecks. Therefore, we introduce Zipkin, which is mainly used for distributed link tracking, and implements a plug-in for Span collection in API Gateway, while the back-end service is implemented through open source middleware.

The above figure is a current overall architecture diagram, the bottom layer is the K8S cluster, above the deployment of Kube-DNS,Consul for service registration discovery and configuration management, and then our hierarchical micro-service system, on the right are some auxiliary management systems.

V. Summary

The above is the whole practice process of micro-service based on Docker and Kubernetes. We have done 9 important things in the process of implementing micro-service, which simplifies the operation process and improves the work efficiency. Getui designs and implements its own micro-service framework, completes the containerized deployment of micro-services, develops the API gateway, and manages the service registration and configuration based on Consul, orchestrates the containers with Kubernetes, reconstructs the service registration and discovery system based on Service and Kube-DNS, builds its own configuration center, optimizes the log service, and implements Zipkin link tracking.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.