Shulou(Shulou.com)06/02 Report--

[technology Salon 002] data Center: construction practice of Credit Agile data Center | Credit Technology Salon will be broadcast live online at 8: 00 p.m. on May 23. Click to sign up.

LAIN is an open source PaaS platform developed by big data Innovation Center of Yixin Company. In the financial context, LAIN is a cloud platform designed to liberate the productivity of teams and lines of business. LAIN provides a unified testing and production environment for all teams of Yixin big data Innovation Center, simplifies the process of service deployment and launch, and reduces the complexity of system management by operation and maintenance personnel.

I. Design concept and problem solving

LAIN standardizes the development, testing and online workflow of an application, and provides an overall solution to devops problems such as container orchestration, authority control, SDN, flow management, monitoring alarm, backup, log and so on.

On LAIN, application is a basic concept. Developers of an application only need to define a lain.yaml to define the way the application is compiled and run, which is very intrusive to the application code. LAIN is based on container technology, faces a variety of technology stacks, and naturally isolates the dependencies of systems and applications.

When LAIN users create an application (service), they can register the application on LAIN, and the current user automatically becomes the maintainer of the application and has the right to further operate the application. The environment for building applications requires docker and lain command-line tools. For convenience, we have created a vagrant box, namely lain-box. When building an application, in addition to the engineering code, you also need a Docker image as the basic image, that is, the compiled environment. If it is a binary project, such as golang, you can replace a bottom at run time, otherwise the build image will be used as a release image. Once you are ready to mirror and compile / run the script, you can edit the lain.yaml.

Specifically, LAIN solves the following four problems:

1. The overall solution to the devops problem under application development.

common problem

In the face of users' application-level development is only the tip of the iceberg, under this organic room, network, server, system management, operation and maintenance management, monitoring, alarm, log and so on a series of work, and this part of the work may be more complex than application-level development using IaaS to solve server procurement and shelf problems, but still need a strong devops team to be responsible for the above affairs Otherwise, infrastructure can easily become a bottleneck in development, and the more it drags on, the more difficult it is to solve the above work, which may be homogenized for every product, but accompanied by customization, which will take a lot of time to do these repetitive tasks.

How does Lain do it?

The lain cluster can be built directly on the almost naked IaaS or server, and the underlying resource operations of the cluster, such as online expansion and reduction, integrate the good overall practice of operation and maintenance deposited by the industry, and provide an overall solution for this large piece of work under the iceberg. The complicated system management and operation and maintenance management behaviors are encapsulated into more easy-to-use toolkits, which greatly simplifies most of the system work. Reduce the technical threshold of daily maintenance and manpower requirements to integrate homogenized work together to avoid repetitive work out of the box with a variety of management components, including deployment, expansion, monitoring, alarm, log and other aspects. There are also complimentary applications, including mysql,redis 's cluster service 2, which standardizes the workflow of application development and is supported by appropriate SCM

common problem

In individual developers and startup organizations, good workflow is hardly mentioned. However, in the process of gradual development, the technical debt left over will more and more affect the efficiency and quality of development and deployment. Irregularities in design, development and deployment behavior will cause all kinds of problems.

How does Lain do it?

The solution that provides the local development environment provides the SDK / CLI toolchain of the local development process, which makes the development and build process embedded in the solution implicitly provides SCM support, restricts the developer's development and release behavior 3, improves the overall resource utilization, and optimizes the redundant resource pool.

common problem

In the traditional case of planning the resource pool according to the product line, each product will be reserved with exclusive resource pool and equipped with redundancy for disaster preparedness and service sudden traffic, but the resource demand type of each product line is different, and the type of redundancy is also different. can not be shared, resulting in a large number of repetitive redundancy, low resource utilization due to server resource redundancy, capacity expansion, and resource migration operations are more complex. High time consumption and high risk

How does Lain do it?

Through the resource isolation and control of container technology, we can realize the secure and non-interactive mixed deployment of multiple technology stacks and applications in the cluster, and carry out redundancy through a unified resource pool. the use of container technology to effectively improve the utilization of resources makes the use of resources form a completely unified form, the cost of capacity expansion and migration is very low, and the operation is easier. 4. TBD: the architecture provides the possibility and solution of service governance. 2. Characteristics

At the application level, LAIN also has the following characteristics:

1. Define applications based on configuration files

On existing applications, you only need to add a configuration file lain.yaml to define the compilation and execution of applications in lain clusters with very low intrusiveness to application code.

2. SDN network security isolation

Using the open source calico (https://github.com/projectcalico/calico) project) to build an efficient SDN network interworking within applications the network between applications is isolated by default and explicitly states the mutual access of services between applications

3. Support diversified technology stack based on container technology.

Use open source docker project to build container cloud extension encapsulation Dockerfile, use custom yaml format for application cluster definition only need to conform to the simplest lain cluster runtime interface, freely choose base image container technology, naturally support isolation system and application dependency lain SDK / CLI and optional ci components support code version and corresponding relationship between image and run-time image can be completely customized and isolated

4. Apply online capacity expansion and reduction

Using open source swarm scheduling application deployment deep encapsulation swarm docker API, self-developed cluster controller (deployd) and application controller (console) directly support user API call for container instance expansion, and capacity reduction directly supports user API call for container single instance resource expansion and CPU,MEM)

5. Node online capacity expansion and reduction

The server node (NODE) of the cluster using the open source ansible (https://github.com/ansible/ansible) development cluster management operation and maintenance kit) is compatible with physical servers, virtual machines, and public cloud servers in the same C segment. The cluster management toolkit supports add NODE and remove NODE instructions to quickly expand and reduce the capacity of the underlying resources.

6. Automatic service maintenance and disaster recovery

Self-developed cluster controller (deployd) container instance-level service inspection and maintenance, automatic migration and service recovery based on virtual ip automatic drift entry load balancer HA advanced API support service customization migration

7. Internal service dependency and discovery mechanism

Cluster support Service / Resource mechanism overall service application private Service (i.e. Resource) service application cluster supports special service application types and resource application types Service / Resource based on DNS service discovery mechanism programmable service/resource load balancer provides available RoundRobin type load balancer by default in lain.yaml

8. Unified authentication

Cluster self-developed Unified Authentication component (sso) supports multiple authentication methods of oauth3

9. Unified management of virtual ip and load balancer

Support the registration of virtual ip and application proc, apply registrable virtual ip to provide external services based on etcd lock virtual ip drift mechanism, and use load balancer to implement HA.

10. Automatic configuration of web load balancer

Use open source nginx and tengine (watcher developed by the load balancer of https://github.com/alibaba/tengine) encapsulating web services) to detect the overall runtime data of cluster applications, automatically generate configurations for web services to obtain the time of runtime changes, and determine whether configuration changes are required. The rendering of the configuration triggers the reload to take effect

11. Systematic log collection of clusters

Using open source heka (https://github.com/mozilla-services/heka) with docker configuration and rsyslog encapsulating cluster log collection default collection application stdout / stderr log collection support application explicit declaration of landing file logs to be collected support application explicit declaration of structured monitoring data log custom detection web service load balancer nginx log collection and data statistics

12. Private docker registry and authentication mechanism

Use open source docker registry to encapsulate private registry application integration to support cluster private unified authentication mechanism customized to support optional moosefs storage backend or Ceph storage backend

13. Configure encrypted storage for application

Use the application private configuration encrypted storage component encapsulated by the open source library to integrate the sso component to achieve user management and privilege isolation to inject the configuration during the application runtime phase

14. Localized development environment

Using open source vagrant, free centos and virtualbox to organize a unified localized development environment and even support local use of the above tool chain bootstrap to create a lain local cluster

15. Application deployment, operation and maintenance API and corresponding CLI client

The construction, release, deployment, operation and maintenance of the application are all provided by the components of the cluster. API is used to encapsulate the above API again using lain SDK / CLI, providing users with a good operation interface to integrate the unified authentication of the cluster, user management and authority isolation.

16. Cluster Management CLI

Using the open source ansible development cluster management operation and maintenance toolkit again encapsulates the ansible call into a simple CLI to make the operation more convenient, including adding nodes, removing nodes, migrating applications, cluster health check and so on.

17. Standardized development workflow

Based on the above components, SCM is carried out according to the one-to-one correspondence between code and image, and the release management of the image is carried out locally by using lain SDK / CLI and optional ci components to build and release, which will naturally standardize the development of workflow workflow. The core unit of the workflow is the image, and lain cli encapsulates the generation, update, push, deployment, and operation and maintenance of the image.

18. Optional cluster systematic backup and recovery (backupd + moosefs)

Using open source moosefs as the distributed storage backend supports explicit declaration of volume backup requirements and policies in lain.yaml, and hooks that sets backup policies to support specified backup recovery

19. Optional cluster log query component (kafka + elasticsearch + kibana)

Using open source kakfa, elasticsearch,kibana to build externally dependent Kafka cluster and elasticsearch cluster, encapsulating cluster optional component libanarebellion cluster log collection component supports sending all logs to the above external dependency kafka supports conditional combination query of cluster application log and web load balancer log on libana

Optional series of preset applications

MySQL Services (https://github.com/laincloud/mysql-service)MySQL Resources Redis Services-SM (https://github.com/laincloud/redis-service-sm) III, system Architecture 1, physical View

From a physical point of view, each lain cluster is made up of one or more network interconnected nodes (Node).

Each node can be assigned a different label for node selection during container scheduling.

In the current implementation, all nodes are required to be behind the same router.

2. Logical view

From a logical point of view, a lain cluster is composed of multiple applications, and the network between applications and applications is isolated from each other (through SDN technology).

Each application consists of multiple Docker containers, each of which may run on different nodes.

Application developers can define multiple containers (called proc) in an application, and each proc can be specified to run multiple copies on the cluster, each of which is a container, called proc instance. The Lain cluster will try its best to ensure that the specified number of containers are running. If a container crash or node fail occurs, the cluster will attempt to restart the container or migrate the container between nodes.

3. System architecture design diagram

The goal is to build an architectural diagram that can be deepened layer by layer.

General drawing

Node

4. Workflow

GitHub address: https://github.com/laincloud

White Paper: https://laincloud.gitbooks.io/white-paper/content/

Source Yixin Institute of Technology

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.