Shulou(Shulou.com)06/02 Report--

Gitlab has been used internally by UCloud to manage code for a long time. Although Gitlab is excellent as an open source platform, we are not very satisfied with the agility it provides for CI/CD, and the code release cycle in internal practice still needs to be calculated on a daily basis. To this end, we have created an internal container service platform based on Kubernetes (called KUN) to host internal services and connect Gitlab to the KUN platform, so as to take advantage of the cloud native advantages of Kubernetes to achieve better CI/CD results. Within one year of operation of the system, Gitlab's Pipeline has been triggered 994 times, and about 20000 + times of Job has been executed, and a total of 7,000 times of deployment has been made in the test environment and formal environment, that is, about 20 times a day. Here are some practical experiences from the project.

Our goal for CI/CD

CI is Continuous Integration (continuous integration), which means that the code must pass the automated test before it is integrated into the backbone. As long as one test case fails, it cannot be integrated. The aim is to allow the product to iterate quickly while maintaining high quality.

CD has two meanings:

Continuous Delivery, continuous delivery, means that any changes are validated and can be implemented and deployed to a production environment at any time.

Continuous Deployment, continuous deployment, is the higher phase of continuous delivery, which means that any modified content is validated and automatically deployed to the production environment.

The difference between the two is whether it is automatically deployed to the production environment. For UCloud, the latter, that is, continuous deployment, must be targeted.

Gitlab branch management

The Gitlab branch management model we adopted has not changed before and after being connected to the KUN platform, so it is briefly described as follows:

Master: the main branch, the code has been verified. Create a tag from master for Release.

Dev: the main R & D branch, which is used to merge the feature branch and the patch branch, in this branch.

Temporary branch:

Feature branch: used to develop a feature

Patch branch: used to repair online bug.

Here is an example to introduce the CI/CD development process. StepFlow is a visual workflow product newly released by UCloud. Through StepFlow users can flexibly and easily define their own workflow and quickly realize business functions. The whole StepFlow system consists of several modules, all of which are deployed on the Kubernetes cluster.

In the example, we need to develop a feature called optimize-allocate:

The development process is as follows:

First, an Issue number 80 is created on Gitlab to follow up the feature of this optimize-allocate

Create a new branch from the dev branch, named feature/80-optimize-allocate, and develop on it

When the development is completed on feature/80-optimize-allocate and commit is carried out, static test, unit test, Review and other Pipeline Job will be triggered.

After the test passes, create a merge request from feature/80-optimize-allocate to dev, which is reviewed by the person in charge. After the audit is passed and the merge is successful, Pipeline Job such as static test, unit test, image construction, image deployment and integration test will be triggered.

After the test passes, create a mergerequest from dev to master, which is reviewed by the person in charge. After the audit is passed and the merge is successful, the person in charge creates tag v1.1.1, and then triggers Pipeline Job such as static test, unit test, image build, image deployment, integration test, etc.

Note: the version number tag has a command specification, v {x}. {y}. {z} represents v {major version}. {minor version}. {minor revision}

Gitlab CI/CD Pipeline

After Gitlab 8.0, Gitlab-CI is integrated and enabled by default, which can provide certain CI capabilities. We build a continuous integration pipeline with three levels of Pipeline, Stage and Job that need to be designed.

Pipeline

Gitlab detects the. Gitlab-CI.yml file under the root directory of a project, in which the user can define CI/CD Pipeline, and a Pipeline represents the entire process of CI/CD. When the code changes (such as push, tag, merge, etc.), a Pipeline will be triggered to run.

Stage

A Pipeline contains multiple Stage, such as "static check", "unit test", "build image", and so on, which are executed sequentially one after another.

Job

Each Stage can contain multiple Job, and all Job of the same Stage execute at the same time. Each Job needs to specify several important attributes such as image, stage, tags, service, etc.

In the StepFlow example, the Pipeline design for the feature to be developed is as follows, including static checks, unit tests, and the last two manual Code Review:

When it needs to be released to a public test environment (similar to a pre-release environment), design another Pipeline, including: perform a complete static check, unit test, pre-release image build, pre-release deployment, pre-release integration test.

After merging master, another Pipeline of prod environment is triggered, including static check, build of production environment image, deployment of production process, and final integration test:

Gitlab Runner

We use Gitlab Runner in conjunction with Gitlab-CI to run the specific Job. Gitlab Runner Kubernetes Executor provides the ability to run Job in Kubernetes. The working principle is as follows:

Gitlab Runner needs to be deployed and registered with Gitlab beforehand

When the code is updated, Gitlab informs runner to run Job according to the user's configuration

Gitlab Runner uses an image to create a Pod, and then runs some commands in it

Gitlab Runner tells Gitlab about the whole running process and the running result.

Kaniko integration and transformation: building a Docker image in a container

In order to use CI/CD to turn the code into a service that ends up running in Kubernetes, an essential step is to build the container image. Because CI Job itself runs as a container, you need to build an Docker image in the container.

Existing methods include mounting Docker Socket on Host into Pod (Docker Socket Mounting), or starting another Docker Daemon in Pod (Docker-in-Docker). However, the former has security risks, while the latter requires Pod to have privileges, neither of which is suitable.

Kaniko (https://github.com/GoogleContainerTools/kaniko) is an open source tool for Google that allows you to create docker images in a docker container without any privileges.

However, native Kaniko images cannot be integrated with Gitlab CI due to the lack of necessary tools. To do this, we modified the Kaniko image and added the entire busybox toolkit to enable it to run as a CI Job. Then you can dock the Gitlab to the internal container service platform KUN. The code example is as follows:

Use Docker:$ cd / path/to/project & &\ docker build-t uhub.service.ucloud.cn/myimage:0.0.1-f deploy/Dockerfile & &\ docker push uhub.service.ucloud.cn/myimage:0.0.1# use Kaniko:$ / kaniko/executor-c / path/to/project-f deploy/Dockerfile-d uhub.service.ucloud.cn/myimage:0.0.1

KUN+Gitlab: Kubernetes-based CI/CD process

The entire process of CI/CD in KUN is shown in the figure above. As we can see from the figure, the CI part is a cyclic process of unit testing, pre-release deployment, integration testing, Debug, and code submission. The pre-released deployment in the CI process is realized through the CD system. One of the steps of CI is to generate the deployment file, and then submit it to the CD backend system according to the project, resource set, version and other parameters. The CD system provides a page entry, and when the deployment file is pushed to the CD system, the user can find the corresponding deployment file on the page. If you need to deploy, click the "deploy" button on the page to achieve deployment.

YAML Editor

For the convenience of users, we have developed a special YAML editor for KUN. Compared with writing YAML with an ordinary text editor, it can provide intelligent template completion, search and replacement and other capabilities.

The figure above shows a use case. The features currently supported by the page editor are:

Snippet: template completion. When users edit a document, they will prompt for relevant templates. You can choose to enter templates directly.

Hover: the user places keywords with the mouse, prompts the meaning of keywords and links to official documents

Search replacement: use ⌘ + F/Ctrl+F to open the page search support pages to query directly

Deploy the system

In order to deploy services to Kubernetes in Gitlab CI Job, we have developed a deployment system. In the last step of CI Pipeline, the user generates a YAML file, defines the resources that need to be deployed to the Kubernetes, and then uses a tool image provided by the deployment system, which calls the API of the deployment system and submits the YAML to the deployment system. Next, the deployment system will use the user's privileges and call Kubernetes API to achieve the actual deployment.

At present, the deployment system mainly consists of two parts:

1. Resource set Management:

A resource set is a collection of one or more resources under a user's project. Resources refer to the object description files of Kubernetes, usually YAML files.

Resource sets are divided into multiple versions, and different versions correspond to different description files of resources.

The user can select a version and then specify the cluster to perform the deployment.

two。 Deployment task

The user generates a job for each deployment, which is the deployment task.

After the deployment, users can view the deployment task log directly on the task details page.

When all the above work is completed, Gitlab can be well embedded in the KUN platform, and the modules running on Kubernetes can run in an orderly manner like gears, thus achieving good results on CI/CD.

The StepFlow project mentioned earlier has implemented such a set of CI/CD from the very beginning, and the efficiency has been greatly improved. Each Job takes about 3 minutes to compile, and the other Job takes less than 1 minute.

Summary

CI/CD is an essential part of providing high-quality Internet services. Gitlab and Kubernetes are both excellent open source software. On this basis, according to our own actual situation, we combine them to create a set of efficient CI/CD platform, strive to improve the efficiency of research and development, and provide users with better services.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.