How to implement end-to-end DevOps pipelining with GitLab 01/17 Update SLTechnology News&Howtos

How to implement end-to-end DevOps pipelining with GitLab

2025-01-17 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Servers >

Shulou(Shulou.com)06/01 Report--

How to implement GitLab end-to-end DevOps pipeline practice, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain in detail for you, people with this need can come to learn, I hope you can get something.

Realize the project end-to-end delivery practice based on Gitlab, from the beginning of requirements development to the delivery pipeline to achieve application release. The workflow for each project team is different, and the workflow in this document is configured according to the previous project team work model. Focus on the implementation of the technology, and the workflow can be defined according to the situation of your own team.

Source: http://www.idevops.site

1. Overall planning and design

Create issue-- > create feature branch-- > feature branch submission pipeline-- > merge branch pipeline-- > publish branch pipeline

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

Create an issues associative property branch (the branch of the feature that begins with a number is the property branch)

The feature branch submits the code, triggering the submission pipeline (build verification and deployment to the feature environment)

The feature environment verification is complete and merged into the RELEASE branch. (trigger the merge pipeline to scan the code, and the pipeline can only be merged successfully)

RELEASE Branch Manual release (UAT,STAG,PROD)

After the production release, the RELEASE branch is merged into the Master branch, and the Release tag is created based on the master branch.

two。 The preparatory work of the demand part

Create milestones

Create issue, associate milestones

Create the corresponding feature branch based on the issue name

3. Pipeline preparation work

You can also use the previous java project directly

Github: https://github.com/zeyangli/gitlabci-cidevops-java-service

Prepare the template library

Prepare the available runner and install and deploy the runner according to the previous content.

Chart: https://github.com/zeyangli/gitlabci-runner-chart-k8s

Configure the project CI file

4. Submit an assembly line design

The developer submits the code in the feature branch, triggers the submission pipeline for code verification and publishes it to the feature environment for verification (the release can be controlled manually).

Phase: compile, test, scan, build image, upload image, publish feature environment

Property environment: the naming convention is the project name-ID- branch name, and each feature branch is published to the corresponding property environment.

Image name:

Registry.cn-beijing.aliyuncs.com/cidevops/cidevops-java-service:3-input-error-a486c590-834

Ingress domain name:

"http://${CI_COMMIT_REF_NAME}.${CI_PROJECT_NAMESPACE}.${CI_PROJECT_NAME}.devops.com"Build phase

Define build job templates and parameterize build commands.

# # build related jobs # # .build: stage: build script:-${BUILD_SHELL}

The build job template is introduced in template, which declares the MAVEN_IMAGE variable to define the image name because it is built using a container. Because the build directory was previously persisted to the build environment, defining the GIT_CLONE_PATH parameter enters the specified build directory operation. GIT_CHECKOUT settings globally there is no need to download code repeatedly for each job. BUILD_SHELL defines the commands required for the build. Defining variables is flexible enough to be suitable for scenarios with different packaging commands for different projects.

Include:-project: 'cidevops/cidevops-newci-service' ref: master file:' jobs/build.yml' variables: # # Global configuration GIT_CLONE_PATH: $CI_BUILDS_DIR/builds/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/$CI_PIPELINE_ID GIT_CHECKOUT: "false" # # dependent container image BUILD_IMAGE: "maven:3.6.3-jdk-8" # # build test parameter MAVEN_OPTS: "- Dmaven.repo.local=/home/gitlab-runner/ci-build-cache/maven" BUILD_SHELL: 'mvn clean package-DskipTests-- settings=./settings.xml' # # build command # # run phase stages:-build cache: paths:-target/ # Jobs Configure # # # build job build: variables: GIT_CHECKOUT: "true" image: ${BUILD_IMAGE} extends: .build

Define the build job and set the job variable GIT_CHECKOUT: "true" indicates that the code needs to be downloaded. The default build is the first job in our pipeline, so it must be set to download the code, otherwise the build fails. Variables in the job take precedence over the global. Image defines the image we want to use, and the image tag can be removed if it is run in non-container mode. The rest of the configuration integrates all the template jobs. Build.

Test stage

What is defined here is the unit test that takes place after running the compilation. Maven projects are generally mvn test,npm projects, generally npm run test and so on. The instructions for running unit tests in different projects do not work, and all other parts are similar. Take the maven project as an example. Start designing unit tests for the maven project.

Edit the jobs/test.yml file to define the test job template. (subsequent automated tests and other test-related jobs are placed in this file)

# Unit test. Test: stage: test script:-$TEST_SHELL artifacts: reports: junit: ${JUNIT_REPORT_PATH}

Edit the template file and add the import test job template.

Include:-project: 'cidevops/cidevops-newci-service' ref: master file:' jobs/build.yml'-project: 'cidevops/cidevops-newci-service' ref: master file:' jobs/test.yml'

Add variable definitions to the template file.

Variables: TEST_SHELL: 'mvn test-- settings=./settings.xml' # # Test Command JUNIT_REPORT_PATH: 'target/surefire-reports/TEST-*.xml' # # Unit Test report # # Test Job test: image: ${BUILD_IMAGE} extends: .test before_script:-ls-ls target/

Code scanning phase

Jobs/code_analysis.yml

# # Code scanning # # .code _ analysis: variables: GLOBAL_PROJECT_ARGS: "- Dsonar.projectKey=$ {CI_PROJECT_NAME}-Dsonar.projectName=$ {CI_PROJECT_NAME}-Dsonar.projectVersion=$ {CI_COMMIT_REF_NAME}-Dsonar.projectDescription=$ {CI_PROJECT_ TITLE} "GLOBAL_SERVER_ARGS:"-Dsonar.ws.timeout=30-Dsonar.links.homepage=$ {CI_PROJECT_URL}-Dsonar.host.url=$ {SONAR_SERVER_URL}-Dsonar.login=$ {SONAR_SERVER_LOGIN}-Dsonar.sourceEncoding=UTF-8 " GLOBAL_MR_ARGS: "- Dsonar.pullrequest.key=$ {CI_MERGE_REQUEST_IID}-Dsonar.pullrequest.branch=$ {CI_MERGE_REQUEST_SOURCE_BRANCH_NAME}-Dsonar.pullrequest.base=$ {CI_MERGE_REQUEST_TARGET_BRANCH_NAME}-Dsonar.gitlab.ref_name=$ {CI_COMMIT_REF_NAME} -Dsonar.gitlab.commit_sha=$ {CI_COMMIT_SHA}-Dsonar.gitlab.project_id=$ {CI_PROJECT_PATH}-Dsonar.pullrequest.gitlab.repositorySlug=$ {CI_PROJECT_ID} "MULTI_BRANCH_ARGS:"-Dsonar.branch.name=$ {CI_COMMIT_REF_NAME} "stage: code_analysis script:-echo $ {GLOBAL_PROJECT_ARGS} ${GLOBAL_SERVER_ARGS} ${SONAR_SCAN_ARGS} ${GLOBAL_MR_ARGS} # sonar-scanner $GLOBAL_PROJECT_ARGS $GLOBAL_SERVER_ARGS $SCAN_JAVA_ARGS-| if [$CI_PIPELINE_SOURCE = = 'merge_request_event'] then echo "sonar-scanner ${GLOBAL_PROJECT_ARGS} ${GLOBAL_SERVER_ARGS} ${SONAR_SCAN_ ARGS} ${GLOBAL_MR_ARGS} "sonar-scanner ${GLOBAL_PROJECT_ARGS} ${GLOBAL_SERVER_ARGS} ${SONAR_SCAN_ARGS} ${GLOBAL_MR_ARGS} else echo" sonar-scanner ${GLOBAL_PROJECT_ARGS} ${GLOBAL_SERVER_ARGS} ${SONAR_SCAN_ARGS} "sonar-scanner ${GLOBAL_PROJECT_ARGS} ${GLOBAL_SERVER_ARGS} ${ SONAR_SCAN_ARGS} ${MULTI_BRANCH_ARGS} fi. Get _ analysis_result: stage: get_analysis_result script:-| SONAR_REPORT_URL=$ (grep "ceTaskUrl". Scannerwork/report-task.txt | awk-F ='{OFS= "=" Print $2 $3}') echo ${SONAR_REPORT_URL} for i in {1.. 10} do curl-k-u "ee2bcb37deeb6dfe3a07fe08fb529559b00c1b7b": "${SONAR_REPORT_URL}-o sonar_result.txt-s grep'" status ":" SUCCESS "'sonar_result.txt & & SONAR_SCAN_RESULT='SUCCESS' if [${SONAR_SCAN_RESULT} = = 'SUCCESS'] then echo "${SONAR_SCAN_RESULT}" SONAR_SCAN_RESULT=SUCCESS break Else SONAR_SCAN_RESULT='ERROR' echo "get the result information for the $I time, not a successful status, sleep for 10 seconds!" Cat sonar_result.txt sleep 10 fi done

Pipeline template

Templates/pipeline.yml

Build Mirror Phase

Jobs/build.yml

.build-docker: stage: buildimage script:-docker login-u $CI_REGISTRY_USER-p $CI_REGISTRY_PASSWD $CI_REGISTRY-docker build- t ${IMAGE_NAME}-f ${DOCKER_FILE_PATH}. -docker push ${IMAGE_NAME}-docker rmi ${IMAGE_NAME}

Telmplate/pipeline.yml

K8S deployment phase

Jobs/deploy.yml

# # Application Publishing # # publish using kubectl image. Deploy _ K8s: stage: deploy script:-echo $KUBE_TOKEN-kubectl config set-cluster my-cluster-- server=$ {KUBE_URL}-- certificate-authority= "${KUBE_CA_PEM_FILE}"-kubectl config set-credentials admin-- token=$ {KUBE_TOKEN}-ls-a-sed-I "sworn publication namespaceworthy publication ${" NAMESPACE} # g "deployment.yaml-sed-I" slots loaded appnametically loaded with ${APP_NAME} # g "deployment.yaml-sed-I" sacks filled with containerports loaded with ${CONTAINER_PORT} # g "deployment.yaml-sed-I" sacks loaded nodeportably loaded with ${NODE_PORT} # g "deployment.yaml-sed-I" sacks filled with imagenameplates such as ${IMAGE_NAME} # g deployment.yaml -sed-I "deployment.yaml sed # g" deployment.yaml-cat deployment.yaml-"kubectl create secret docker-registry ${ENV_URL} # g" deployment.yaml-cat deployment.yaml-"kubectl create secret docker-registry ${ENV_URL} # g" deployment.yaml-sed-I "deployment.yaml-sed-I" deployment.yaml-cat deployment.yaml-"deployment.yaml-cat deployment.yaml -" _ NAME}\-- docker-server=$ {CI_REGISTRY}\-- docker-username=$CI_REGISTRY_USER\-- docker-password=$ {CI_REGISTRY_PASSWD}\-- docker-email=test@test.com-n ${NAMESPACE} | | echo 'secrets already exists' "- kubectl apply-f deployment.yaml environment: name:" ${CI_COMMIT_REF _ NAME} "url:" http://${CI_COMMIT_REF_NAME}.${CI_PROJECT_NAMESPACE}.${CI_PROJECT_NAME}.devops.com" on_stop: rollout_k8s when: manual # # rollback. Rollout _ K8s: stage: rollout script:-rm-rf $HOME/.kube-kubectl config set-cluster my-cluster-- server=$ {KUBE_URL}-- certificate-authority= "${KUBE _ CA_PEM_FILE} "- kubectl config set-credentials admin-token=$ {KUBE_TOKEN}-kubectl rollout history deployment ${APP_NAME}-n ${NAMESPACE}-kubectl rollout undo deployment ${APP_NAME}-n ${NAMESPACE} when: manual environment: name:" ${CI_COMMIT_REF_NAME} "action: stop

Templates/pipeline.yml

5. Pipeline trigger control

Using workflow:rules for pipelined control, we will use Pipeline variables to restrict conditions through variables.

Predefined variable reference documentation: https://docs.gitlab.com//12.9/ee/ci/variables/predefined_variables.html

Variable matching syntax: https://docs.gitlab.com//12.9/ee/ci/variables/README.html#supported-syntax

Re2 syntax: https://github.com/google/re2/wiki/Syntax

Exclude the pipeline of the new branch

Run the pipeline and you will find that the CI_COMMIT_BEFORE_SHA of all newly created branches is 40 zeros.

$export

Declare-x CI_BUILD_BEFORE_SHA= "000000000000000000000000000000000000"

Declare-x CI_COMMIT_BEFORE_SHA= "000000000000000000000000000000000000"

# # pipelined control workflow: rules:-if: $CI_COMMIT_BEFORE_SHA = = "000000000000000000000000000000000000000000" when: never-when: always

How do you match the feature branch and the version branch?

# $CI_COMMIT_REF_NAME = ~ /\ dMushroom / # $CI_COMMIT_REF_NAME = ~ / ^ RELEASE-*/ | | merge the pipeline and then verify the construction

As you can imagine, if it is a team that is just beginning to pay attention to code quality, it cannot avoid code quality threshold failure. It is normal for errors to occur at the initial stage of improvement, but if the quality threshold is strictly configured at the initial stage, this will result in an error every time the code is submitted. So we can appropriately release the code scanning of the pipeline (that is, the pipeline does not check the quality threshold for the time being).

There is no way to know the exact quality of the code without scanning, so we prepare the pipeline to scan but not check the quality threshold, while the merge pipeline will display the code quality in the comments area. In a case like this, we can set the pipeline to be successful before we can merge.

The default is that the commit triggers the pipeline to run, while setting "merge after pipeline success" will check whether the status of the last commit of the original branch is success, and if it is success, run the merge. We configure the pipeline to validate the code when a merge request occurs.

# # Pipeline control workflow: rules:-if: $CI_MERGE_REQUEST_ID6. Deployment pipeline practice

We also store the deployment files of the application in the code base for management, and the configuration files of each application may be inconsistent in different environments. All are divided into three configuration files: deployment-uat.yml, deployment-stag.yml, deployment-prod.yml

Jobs/deploy.yml

# # Application Publishing # # publish using kubectl image. Deploy _ K8s: stage: deploy script:-echo $KUBE_TOKEN-kubectl config set-cluster my-cluster-- server=$ {KUBE_URL}-- certificate-authority= "${KUBE_CA_PEM_FILE}"-kubectl config set-credentials admin-- token=$ {KUBE_TOKEN}-ls-a-sed-I "sworn publication namespaceworthy publication ${" NAMESPACE} # g "${DEPLOY_FILE}-sed-I" slots loaded appnametically loaded with containers ${APP_NAME} # g "${DEPLOY_FILE}-sed-I" sworn containers loaded with containerports ${CONTAINER_PORT} # g "${DEPLOY_FILE}-sed-I" sworn cargo nodeportables installed ${NODE_PORT} # g "${DEPLOY_FILE}-sed-I" s#__imagename__# ${IMAGE_NAME} # g "${DEPLOY_FILE}-sed-I" slotted packing procedures for example, "${CI_ENVIRONMENT_SLUG} # g" ${DEPLOY_FILE}-sed-I "slicing procedures for packing packages" ${CI_PROJECT_PATH_SLUG} # g "${DEPLOY_FILE}-sed-I" slotted packing packages for example ${ENV_URL} # g "{ENV_URL} # g" DEPLOY_FILE}-cat ${DEPLOY_FILE}-"kubectl create secret docker-registry ${APP_NAME}\-docker-server=$ {CI_REGISTRY}\-docker-username=$CI_REGISTRY_USER\-docker-password=$ {CI_REGISTRY_PASSWD}\-docker-email=test@test.com-n ${NAMESPACE} | | echo 'secrets already exists '"- kubectl apply-f ${DEPLOY_FILE} environment: name:" ${ENV_NAME} "url:" http://${ENV_NAME}.${CI_PROJECT_NAMESPACE}.${CI_PROJECT_NAME}.devops.com" on_stop: "${ROLL_NAME}" # # rollback .rollout _ k8s: stage: deploy script:-rm-rf $HOME/.kube- Kubectl config set-cluster my-cluster-- server=$ {KUBE_URL}-- certificate-authority= "${KUBE_CA_PEM_FILE}"-kubectl config set-credentials admin-- token=$ {KUBE_TOKEN}-kubectl rollout history deployment ${APP_NAME}-n ${NAMESPACE}-kubectl rollout undo deployment ${APP_NAME}-n ${NAMESPACE} environment: name: "${ENV_NAME}" action: stop

Templates/pipeline.yml

Include:-project: 'cidevops/cidevops-newci-service' ref: master file:' jobs/build.yml'-project: 'cidevops/cidevops-newci-service' ref: master file:' jobs/test.yml'-project: 'cidevops/cidevops-newci-service' ref: master file:' jobs/codeanalysis.yml'-project: 'cidevops/cidevops-newci-service' ref: master File: 'jobs/deploy.yml' variables: # # Global configuration GIT_CLONE_PATH: $CI_BUILDS_DIR/builds/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/$CI_PIPELINE_ID GIT_CHECKOUT: "false" # # dependent container image BUILD_IMAGE: "maven:3.6.3-jdk-8" SONAR_IMAGE: "sonarsource/sonar-scanner-cli:latest" # # Construction Build the test parameter MAVEN_OPTS: "- Dmaven.repo.local=/home/gitlab-runner/ci-build-cache/maven" BUILD_SHELL: 'mvn clean package-DskipTests-- settings=./settings.xml' # # build command TEST_SHELL: 'mvn test-- settings=./settings.xml' # Test command JUNIT_REPORT_PATH: 'target/surefire-reports/TEST-*.xml '# # Unit Test report # # Code scan SONAR_PROJECT_LANG: "JAVA" SONAR_SOURCE_DIR: "src" SONAR_SERVER_URL: "http://192.168.1.200:30090" SONAR_SERVER_LOGIN:" ee2bcb37deeb6dfe3a07fe08fb529559b00c1b7b "SONAR_SCAN_ARGS:"-Dsonar.sources=$ {SONAR_SOURCE_DIR}-Dsonar.java.binaries=target/classes -Dsonar.java.test.binaries=target/test-classes-Dsonar.java.surefire.report=target/surefire-reports "# build an image CI_REGISTRY: 'registry.cn-beijing.aliyuncs.com' CI_REGISTRY_USER:' xxxxxx' # CI_REGISTRY_PASSWD: 'xxxxxxxx.' IMAGE_NAME: "$CI_REGISTRY/$CI_PROJECT_PATH:$CI_COMMIT_REF_NAME-$CI_COMMIT_SHORT_SHA" DOCKER_FILE_PATH: ". / Dockerfile" # deploy application K8s RUN_DEPLOY: "yes" APP_NAME: "$CI_PROJECT_NAME" CONTAINER_PORT: 8081 # NODE_PORT: 30185 NAMESPACE: "$CI_PROJECT_NAME-$CI_PROJECT_ID-$CI_ENVIRONMENT_SLUG" # # pipeline control System workflow: rules:-if: $CI_MERGE_REQUEST_ID-if: $CI_PIPELINE_SOURCE = = 'web'-if: $CI_COMMIT_BEFORE_SHA = = "00000000000000000000000000000000000000000000000000000000000000000000" when: never # # Runtime stages:-build-test-parallel01-get_analysis_result-deploy-rollout cache: paths:-target/ before_script:-export # # # Jobs Configure # build job build: variables: GIT_CHECKOUT: "true" image: ${BUILD_IMAGE} extends: .build # # Test job test: image: ${BUILD_IMAGE} extends: .test before_script:-ls-ls target/ # # Code scan code_analysis: stage: parallel01 image: ${SONAR_IMAGE} extends: .code _ analysis # # get the construction result get_analysis_result: image: curlimages/curl:7.70.0 extends: .get _ analysis_result needs:-code_analysis # # build image build_image: image: docker:latest services:-name: docker:dind stage: parallel01 extends: .build- Docker # # feature Publishing App deploy_feature: variables: DEPLOY_FILE: 'deployment.yaml' ENV_NAME:' feature' stage: deploy image: lucj/kubectl:1.17.2 extends: .deploy _ K8s rules:-if: $CI_COMMIT_REF_NAME = ~ / ^ RELEASE-*/ when: never-if: $CI_COMMIT_REF_NAME = ~ /\ dMurray / when: manual-when : never # # Application rollback rollout_feature: stage: rollout image: lucj/kubectl:1.17.2 extends: .rollout _ K8s rules:-if: $CI_COMMIT_REF_NAME = ~ / ^ RELEASE-*/ when: never-if: $CI_COMMIT_REF_NAME = ~ /\ dMustang / when: manual-when: never # # UAT deploy_uat: variables: DEPLOY_FILE: 'config / deployment-uat.yaml' ENV_NAME: 'uat' stage: deploy image: lucj/kubectl:1.17.2 extends: .deploy _ K8s rules:-if: $CI_COMMIT_REF_NAME = ~ / ^ RELEASE-*/ when: manual-when: never # # UAT Application rollback rollout_uat: stage: rollout image: lucj/kubectl:1.17.2 extends: .rollout _ K8s rules:-if : $CI_COMMIT_REF_NAME = ~ / ^ RELEASE-*/ when: manual-when: never # # STAG deploy_stag: variables: 'config/deployment-stag.yaml' ENV_NAME:' stag' stage: deploy image: lucj/kubectl:1.17.2 extends: .deploy _ k8s rules:-if: $CI_COMMIT_REF_NAME = ~ / ^ RELEASE-*/ when: manual- When: never # # STAG application rollback rollout_stag: stage: rollout image: lucj/kubectl:1.17.2 extends: .rollout _ k8s rules:-if: $CI_COMMIT_REF_NAME = ~ / ^ RELEASE-*/ when: manual-when: never # # PROD deploy_prod: variables: DEPLOY_FILE: 'config/deployment-prod.yaml' ENV_NAME:' prod' stage: deploy image: lucj/ Kubectl:1.17.2 extends: .deploy _ K8s rules:-if: $CI_COMMIT_REF_NAME = ~ / ^ RELEASE-*/ when: manual-when: never # # PROD Application rollback rollout_prod: stage: rollout image: lucj/kubectl:1.17.2 extends: .rollout _ k8s rules:-if: $CI_COMMIT_REF_NAME = ~ / ^ RELEASE-*/ when: manual-when: never7. After the release is completed. Merge the version branch into the master branch

two。 Create a version label based on a master branch

3. Turn off issues and milestones

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.

*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.