Shulou(Shulou.com)06/02 Report--

one。 Overview 1.1 background

Recently, the migration platform project jointly developed with colleagues wants to carry out containerization transformation, which conforms to the general trend and relies on containerization. The front-end platform of the project is developed using Django, the back-end Restful API is developed using high-performance Web framework Tornado, and the agent is developed using Flask, each taking the advantages of several large Python frameworks.

Previously, the CI/CD test environment used Gitlab CI,master to automatically build and deploy after submitting the merge request. The formal environment manually pulled the release deployment through Jenkins pipeline, the containerization transformation hosted the Jenkins on the Kubernetes, and the Master accepted job requests and dynamically generated slave to complete the job task.

This article records some of the experience encountered in the containerization transformation, may not be enough, the following are personal understanding, the boss does not like to spray, in the use of Kubernetes in the project containerization process, decided that the python project is a little overqualified, but through this transformation, understand a lot of containerization characteristics, constantly improve their IT technology, enrich their skills stack.

1.2 containerization transformation has more advantages: great resource utilization efficiency, maximum extraction and sharing of physical resources, multi-projects can better reflect the multiple advantages of containerization, and save the cost of deploying IT. Faster: start in seconds to achieve faster development iteration and delivery deployment of business systems. Elasticity: the elastic container can be expanded and expanded according to the business load. Convenience: container-based business deployment supports blue-green / grayscale / canary and other releases, rollback, more flexible and convenient. Flexibility: monitor the health status of the underlying node nodes and flexibly schedule to the optimal node deployment. Strong consistency: the container packages the environment and code in a mirror, ensuring strong consistency between the test and production environments. 1.3 containerization transformation requires developers to be familiar with Docker virtualization technology and be proficient in writing Dockerfile. Familiar with kubernetes containerized orchestration system, familiar with the compilation of component resource list. The development needs to take into account the needs of later container orchestration deployment to organize the structure and write code. The deployer needs to be familiar with the meaning of the parameters of the kubernetes resource list and control the architecture from top to bottom as a whole. Consider high availability architecture and rbac security policy, external traffic introduction and later expansion and scaling. two。 Tools 2.1 Cloud native ecology

Once in the cloud, it is as deep as the sea. The picture below shows us a better overall understanding of the primary ecology of the cloud.

2.2 tool application

Share some of the tools and applications used in this project (later, you will have time to write and share each tool application separately)

Code hosting

The Gitlab server does code hosting, and gitlab ci/cd, which can be hosted on the kubernetes cluster later.

Private mirror hosting

Harbor is used for image storage, audit management and image checking, which can be hosted on kubernetes later.

Cluster management

Kubernetes-dashboard deployment, web interface is convenient for various components to view and manage, simple container terminal management, log sharing and viewing.

Rancher deployment, import privatized kubernetes platform to facilitate cluster management and app installation and deployment.

Storage management

Ceph to mgr distributed Cluster web Interface Management

Minio/chartmuseum object storage, convenient for chart storage management.

Integrated release

Jenkins is continuously integrated and released, and can be hosted on the kubernetes cluster at a later stage.

Log monitoring

Efk monitors and manages kubernetes cluster container logs, and f is a powerful monitoring tool for flutend containerization.

Helm warehouse management

Kubeapps adds chart,registry to facilitate helm installation and deployment.

In-container app monitoring

Prometheus + grafana, export in each app to monitor a single app to matric.

The following picture is the navigation page, which shows these tools visually.

three。 3.1.1 the project structure creates a separate config folder for configuration files, which is convenient for later kubernetes to create configmap for resource mapping.

​

If you later deploy as a deployment stateless application, you should create a separate directory for the shared data storage to facilitate mounting on volume later.

​

Create a separate deploy directory for storage under the project result directory

​

For example, the sub-project is divided into creating a Secret where the configmap/deployment/service has pulled the private repository code.

Entrypoint.sh is used as the container entry under the project. Finally, exec "$@" is added to make it easier to add configuration extensions later.

​

3.1.2 the code requires that the configuration file should be written in Yaml language as much as possible, so that it can be loaded into configmap later to facilitate modification.

Because it is convenient for container monitoring in the later stage, Fluentd is used with EL for cluster and application monitoring, and the monitoring container directory is / var/lib/docker/containers/*.log, so you need to output the log to stdout, so if you need to monitor the log, you can direct it to the standard output / standard error output log file.

​

3.2 Architectural requirements 3.2.1 basic resource computing

Use CVM to build and deploy kubernetes clusters to provide computing and memory resources.

Storage

You need to deploy a Ceph distributed storage system using CVM disks to provide underlying storage resources for kubernetes.

The network

LB is required in the front section, and the flannel network is used to proxy the application to the NodePort,kubernetes cluster.

Node communicates through vpc VPC before going to node

Inter-container communication: communication between multiple containers within the same POD, using lo Nic to communicate between POD: POD IP communicates directly with POD IP, POD and Service:POD IP communicate directly with Cluster IPService and external clients of the cluster, ingress, NodePort, Loadbacer3.2.2 traffic introduction needs to be deployed on public ownership / privatization, or bare metal. When you need to plan the previous period in advance, Cloud LB can use its deployment certificate to load certificates at seven layers. If there is no cloud product, you need to consider the introduction of ingress traffic into the cluster to load the certificate. four。 Project section example 4.1 storage class

Use ceph cluster to build storage classes, and use CephFS to solve cross-node mount applications.

Storage class ceph-storageclass.yamlapiVersion: storage.k8s.io/v1kind: StorageClassmetadata: name: ceph-rdbprovisioner: ceph.com/rbdreclaimPolicy: Retainparameters: 10.xx.xx.xx:6789 pool: kube adminId: admin adminSecretName: ceph-admin-secret adminSecretNamespace: kube-system userId: kube userSecretName: ceph-client-secret userSecretNamespace: kube-system fsType: xfs imageFormat: "2" imageFeatures: "layering" ceph Certification ceph-secret.yamlapiVersion: v1kind: Secretmetadata: name: ceph-admin-secret namespace: kube- Systemtype: "kubernetes.io/rbd_" data: # ceph auth get-key client.admin | base64 key: QVFCRitmUmM1c1FxxxxxxxxxxxxxxxxxxxxxxxxHFoQVh7NlRvQ2c9PQ== for shared directories apiVersion: v1kind: PersistentVolumeClaimmetadata: name: go2cloud-api-pvc namespace: defaultspec: storageClassName: "ceph-rdb" accessModes:-ReadWriteOnce resources: requests: storage: 8Gi configuration file mounts apiVersion: v1data: config.yaml: |-- DB_ENGINE: mysql DB_HOST: mariadb-cluster-mariadb- via configmap Master.default.svc.cluster.local DB_PORT: 3306 DB_USER: go2clouduser DB_PASSWORD: go2xxxxxxxxx DB_NAME: go2cxxxxxxxx # Use Redis as cache # Redis configuration Master node connected to replication REDIS_HOST: redis-cluster-redis-ha-announce-0.default.svc.cluster.local REDIS_PORT: 6379 REDIS_PASSWORD: go2cloxxxxxxxx # go2cloud-platform listening port HTTP_LISTEN_PORT: 8088 # callback url API_MIGRATE_SERVER_URL: http://go2cloud-api-service.default.svc.cluster.local:8004 PLATFORM_CALLBACK_URL: http://go2cloud-platform-service.default.svc.cluster.local : 8088kind: ConfigMapmetadata: name: go2cloud-platform-cm4.2 App related Resources go2cloud-platform-deployment.yamlapiVersion: apps/v1kind: Deploymentmetadata: name: go2cloud-platform namespace: defaultspec: selector: matchLabels: # match the label.app name app: go2cloud-platform replicas: 2 template: metadata: labels: app: go2cloud-platform release: latest spec: imagePullSecrets in the template selected below: -name: registry-secret containers:-name: go2cloud-platform image: 10.234.xxx.xxx/go2cloud/go2cloud-plaxxxxx:latest imagePullPolicy: IfNotPresent ports:-containerPort: 8088 protocol: TCP volumeMounts: # must match the name of volumes-name: go2cloud-platform-config mountPath: / data/config readOnly: true resources: Requests: cpu: 250m memory: 520Mi limits: cpu: 500m memory: 1024Mi livenessProbe: tcpSocket: port: 8088 initialDelaySeconds: 20 volumes: # define the name of the logical volume-name: go2cloud-platform-config configMap: # use configmap resources The name name: go2cloud-platform-cm items: # uses the key-key: config.yaml # in configmap to map to the key in configmap to the container to the file name path: config.yaml mode: 0644 go2cloud-platform-service.yamlapiVersion: v1kind: Servicemetadata: name: go2cloud-platform-service namespace: defaultspec: selector: App: go2cloud-platform type: NodePort ports:-name: http nodePort: 30020 port: 8088 targetPort: 8088 protocol: TCPregistry-secret.yamlapiVersion: v1kind: Secretmetadata: name: registry-secrettype: kubernetes.io/dockerconfigjsondata: .dockerconfigjson: ewoJImF1dGhzIjogewoJCSIxMC4yMzQuMi4yMTgiOiB7CgkJCSJhdXRoIjogIllXNWphRzVsZERwWWVIcDRRRGM0T1E9PSIKCQl9Cgl9LAoJIkh0dHBIZWFkZXJzIjogewoJCSJVc2VyxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxDockerfileFROM python:latestLABEL maintainer= "kaliarch" ENV BASE_ROOT= "/ data" ADD. ${BASE_ROOT} RUN pip install-- default-timeout=100-r ${BASE_ROOT} / requirements/requirements.txt\ & & ln-s ${BASE_ROOT} / entrypoint.sh / bin/entrypoint.shEXPOSE 8088/tcpENTRYPOINT ["/ bin/sh", "/ bin/entrypoint.sh"] CMD ["python", "/ data/runserver", "start", "all"] entrypoint.shedding. Reflect on the personal decision that the cloud will become a major trend in the later period, and master containerized layout technology in advance so that the technology stack will not be out of date. It will be a lot faster to make full use of cloud ecological applications to carry out containerization transformation in line with the characteristics of their own projects. Feel that cloud origin is the main trend in the future. Get started as soon as possible and embrace cloud origin.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.