Shulou(Shulou.com)06/03 Report--

This article introduces the "Gitee + Jenkins-based Open Source Project Automation collaboration method tutorial" related knowledge, in the actual case operation process, many people will encounter such a dilemma, then let the editor lead you to learn how to deal with these situations! I hope you can read it carefully and be able to achieve something!

Abstract: today, with the increasingly active concept of open source, more and more people begin to devote themselves to open source and contribute more and more open source projects. With the passage of time, more and more people begin to contribute to open source projects and contribute to open source projects in a certain field. The increase of contributors brings a lot of audit pressure to open source authors. In fact, these open source authors who devote themselves to open source basically do it in their spare time and do not spend much time on open source projects. In order to solve this kind of scenario, this article introduces how to start automated collaboration for your open source projects through Jenkins on Gitee, in order to automate some open source related work and set aside more time to devote more time to the open source cause.

Brief introduction of Gitee & Jenkins

Mayun Gitee is a Git-based code hosting and collaborative development platform launched by Open Source China (OSChina.net) in 2013, which provides code hosting services and complements and promotes with open source Chinese community information, blogs, communities and other sections, hoping to better serve developers and build a more perfect open source ecology. After more than seven years of development, it has become the largest code hosting platform in China.

Jenkins is an open source CI&CD software that automates a variety of tasks, including building, testing, and deployment software, and supports a variety of operating modes, either through the system package, Docker, or through a stand-alone Java program.

Open source project collaboration process collaboration approach

PullRequest (MergeRequest) is generally used to contribute code to open source projects, which generally requires users to Fork the open source repository to their own name, then modify and submit it based on their own repository, and finally submit it to the main repository of the open source project in the form of PullRequest, which means that I request to merge my code into your main warehouse to repair / contribute some code for this open source project. For specific use, please refer to the Gitee official document: https://gitee.com/help/articles/4128

In addition to the regular PullRequest approach, Gitee also introduces lightweight PullRequest (lightweight PR for short), which eliminates the tedious Fork process and quickly submits a PullRequest to open source projects. For example, when I] find expression or spelling errors when reading Readme, I can modify the Readme file directly and automatically generate a lightweight PR when submitting it, which is similar to the error correction function of the knowledge sharing platform. For more information, please see: https://gitee.com/help/articles/4291

It is more convenient than regular PullRequest,Gitee lightweight PR.

Pain point

When the author of an open source project receives the first PullRequest submitted by a user, he must be very excited. After all, this is a kind of recognition for his own project. And according to my personal observation, in general, when the number of Star of open source projects exceeds 100, it is possible to harvest their first PullRequest.

When the number of PullRequest is small, manpower can fully handle it, so what if it is a very popular open source project? The answer is that there will be a lot of enthusiastic users to help submit and contribute code, and these PullRequest are likely to cause a squeeze. As the project evolves, compliance is gradually on the agenda, and specifications for code submission, and even Commit Message submission, may need to be standardized.

At present, the main pain points in open source collaboration are as follows:

Non-standard submission information, such as mixed use of Chinese and English, duplicate submission information, etc.

The submitted code needs to be manually tested to see if it can pass the build

Canonical format checking is required for specific files

Code quality check requires repeated communication and follow-up.

The contributor did not read the contributor agreement or specification in detail.

How to solve

At present, PullRequest Flow can be configured through Gitee + Jenkins to achieve intelligent automatic preprocessing of some scenarios and reduce human maintenance. Gitee provides a Gitee Jenkins Plugin plug-in, which can be linked with Jenkins to automatically check each PullRequest and realize the flow chart:

Let's take a gadget GiteeAutoUpdate (address: https://gitee.com/kesin/GiteeAutoUpdate, hereinafter referred to as GAU) as an example to introduce the specific implementation.

Note: the following sample code and configuration are for the purpose of Demo without any exception or error handling. If you need to put into production, please handle the exception and error to avoid unexpected situations:)

Realize the effect picture:

Automatic configuration practice

GAU is a Webhook server that is pushed to Gitee and automatically updated to other platforms. Through simple configuration, the submission pushed to Gitee can be automatically pushed to other platforms. In order to automate some pre-checking of user-submitted PullRequest, suppose we need to do the following:

Configure PullRequest to be linked with Jenkins. When PullRequest is submitted or updated, Jenkins is triggered to perform pre-check.

After submitting the PullRequest, the user is automatically prompted to read the Contributor Agreement, and then comment on Rebuild for further inspection (you can also say that commenting on Rebuild is regarded as agreeing to the Contributor Agreement)

Check if there are any duplicates in the Commit Message (we require that each Commit Message has a specific meaning)

Verify that the build is successful

Check that the configuration file is in the correct format (targeted checks need to be done for some key information)

Integrate Sonar quality testing tools to detect code

Configuration of Jenkins

Gitee provides the Gitee Jenkins Plugin plug-in, which can be installed in the Jenkins plug-in market search. After the installation is complete, we create a new Free Style project named GAU.

Configure the Git source in Source Code Management as follows:

In Build Triggers, select the following configuration:

And add Gitee WebHook URL to the WebHook of the corresponding warehouse

Webhook Token is generated in the Secret Token for Gitee WebHook of Jenkins. It should be noted here that if your Jenkins is exposed to the public network, you may set the Jenkins to be accessible only if permission is required, then Gitee will not be able to request your WebHook. You need to use the plug-in Role-Based Strategy to configure the permissions of anonymous users to Build. But don't worry, you can combine the newly generated Token to ensure a safe trigger.

Then configure a build phase Build. We select Execute shell and fill in a simple Hello World first.

Finally, configure the postback message after failure in Post-build Actions. Here, we only need to configure the action after failure, because in other cases, we will send it back by ourselves through the script.

Submitting a PullRequest will automatically trigger the build and prompt Triggered by ZokerBot Gitee PullRequest # 1: kesin/GiteeAutoUpdate = > master, and the task execution will type the Hello World we just wrote

For specific configuration, please see plug-in description: https://gitee.com/oschina/Gitee-Jenkins-Plugin#%E6%8F%92%E4%BB%B6%E9%85%8D%E7%BD%AE

Configure automatic prompt messages after PullRequest submission

Sometimes, we need to automatically add a prompt or ask the submitter to take some other actions after the user submits the PullRequest. Here we assume that the user needs to read the notes for the contributor:

# contribution code 1. Please do not repeat the submission information to ensure that each submission information is meaningful. Please add comment 3 for the submitted code. After the code is merged, the ownership will follow the LICENSE### request of the project itself to add synchronization 1. 1. Please make sure that authorized users have been added to the corresponding project 2. Make sure that Webhook has been added to your warehouse 3. Make sure the content of `syncWhitelist` is in accordance with the format.

To ensure that the comments are clean and tidy, we add content to the Issue: https://gitee.com/kesin/GiteeAutoUpdate/issues/I28BNK

So, the welcome message we need should look like this:

For quick implementation, let's write a script giteeCheck using Go to implement this function:

Func welcomeMsg (PRIID string) {params: = make (map [string] interface {}) params ["access_token"] = TOKENif noteNotExist (PRIID) {/ / create first welcome noteparams ["body"] = fmt.Sprintf (`Welcome to submit PR. You need to take the following two steps: 1. [click here to read the Contributor Agreement and Notes] (https://gitee.com/%s/%s/issues/%s) 2. In this PR comment "Rebuild" continues to execute the build integration test status Indicates that you have read and agreed to the Contributor Agreement `, OWNER, REPO, ISSUE) commentUrl: = fmt.Sprintf ("https://gitee.com/api/v5/repos/%s/%s/pulls/%s/comments", OWNER, REPO, PRIID) postForm (commentUrl, params) fmt.Printf (" 0 ") return} fmt.Printf (" 1 ")}

You only need to pass in the current PRIID. In order to identify whether it is submitted for the first time, we can determine whether there has been a comment, so we can add this command during the construction process of Jenkins.

# add welcome msg when PR createdstatus= `/ home/zoker/app/jenkins/scripts/giteeCheck welcome ${giteePullRequestIid}`

Where ${giteePullRequestIid} is the environment variable passed in Gitee Jenkins Plugin, which can be called directly in SHELL to save the return value to status to determine whether it is the first welcome message or a later build.

Check whether the Commit Message submission information is duplicated

Whether the submission information is repeated can be determined according to the difference between the source branch and the target branch. Git provides a way to quickly get the submission information of the difference between the two versions.

Git log HEAD ^ remotes/origin/master-- pretty=format:'%B'

Through the comparison of the number before and after deduplication, we can see whether there is duplicate submission information. The command for this process is as follows:

# check commit msg commitCountA= `git log HEAD ^ remotes/origin/master-- pretty=format:'%B' | sort-r | grep-v'^ $'| wc-l`git log HEAD ^ remotes/origin/master-- pretty=format:'%B' | sort-r | grep-v'^ $'| uniq | wc-l`if [$commitCountA= = $commitCountB]; then commitMsg= "valid" else commitMsg= "invalid" fi

After execution, we save the result in the commitMsg variable for subsequent information to be sent back.

Verify that the build is successful

This step of detection is relatively simple, just execute the corresponding build command, and then determine whether the build exists or not.

# build and check validation go build main.go if [- x gau]; then buildStatus= "valid" else buildStatus= "invalid" fi

Similarly, we save the results in BuildStatus for subsequent information to be sent back.

Check that the configuration file is in the correct syncWhitelist format

The purpose of checking syncWhitelist here is to avoid startup failure due to the failure to read the correct warehouse whitelist information when the program starts. Similar behavior can also be a team-developed inspection tool or a commercial inspection tool. Users check the code format, comments, and whether there is an accidentally uploaded Token. Instead, check the Json format of the file. Similarly, we wrote a script to decode it to ensure that the decoding works properly:

Func checkFile (file string) {projects: = make (map [string] [] string) jsonFile, err: = os.Open (file) if err! = nil {fmt.Printf ("invalid") return} defer jsonFile.Close () jsonParser: = json.NewDecoder (jsonFile) if err: = jsonParser.Decode (& projects); err! = nil {fmt.Printf ("invalid") return} fmt.Printf ("valid")}

Configure to the build command:

# check syncWhitefile validation syncFormat= `/ home/zoker/app/jenkins/scripts/giteeCheck checkfile config/ syncWhitelist` integrate Sonar code quality detection

Sonar (SonarQube) is an open source platform for managing the quality of source code. Sonar is not only a quality data reporting tool, but also a code quality management platform. It is very simple to install and use, as long as it supports Java 11 environment. I will not repeat the installation and use of Sonar here, for more information, please refer to the official document: https://docs.sonarqube.org/latest/

What we need to know is that Sonar consists of two parts: SonarWeb and SonarRunner. SonarWeb is used to collect data and display. SonarRunner is used to analyze warehouse code generation quality data. As long as you execute a command under the corresponding warehouse, you can get the analysis report. We can do some access control based on the heavy data of this report. For example, if you have more than 5 warnings or if there is a serious one, you can reject PullRequest directly. Let the submitter perfect and then submit, can liberate the manpower to a certain extent.

In order to quickly implement, we will just focus on warningCount. As long as warningCount is greater than zero, we will remind the user that there may be risks. After executing the branch command, we can get an API address from the result, which contains some basic data of this analysis:

INFO: Analysis report generated in 50ms, dir size=134 KBINFO: Analysis report compressed in 15ms, zip size=31 KBINFO: Analysis report uploaded in 15msINFO: ANALYSIS SUCCESSFUL You can browse http://127.0.0.1:8089/dashboard?id=taskoverINFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis reportINFO: More about the report processing at http://127.0.0.1:8089/api/ce/task?id=AXYoHjxxxxxxxxDlfFINFO: Analysis total time: 2.788 sINFO:-- -- INFO: EXECUTION SUCCESS

All we need is to get the http://127.0.0.1:8089/api/ce/task?id=AXYoHjxxxxxxxxDlfF address and send it to our script for further processing:

# Sonar check / home/zoker/app/sonar/scanner/bin/sonar-scanner-Dsonar.projectKey=GAU-Dsonar.sources=. -Dsonar.host.url= http://127.0.0.1:8089-Dsonar.login=410e439472a0626bde66aa7564fc3faa9430f7c6 > sonarRes sonarResUrl= `cat sonarRes | grep api/ce | cut-d''- f8`cat 3 / / prevent the SonarWeb from processing the request.

At this point, several test items have been configured, and the status syncFormat commitMsg buildStatus sonarResUrl and so on we need have been saved in variables, and then we will send these data back to PullRequest's comments.

Constructed the result feedback.

The result postback is nothing more than returning a series of results to the corresponding PullRequest in a fixed format through the comment API of PullRequest. We process the data through script and return specific information according to the specific situation:

Func resultBack (args [] string) {var a, b, c, d = "success", "success" status: = make (mapping [string] string) status ["valid"] = ": white_check_mark:" status ["invalid"] = ": X:" / / sonar warningCountsonarRes, _: = http.Get (args [3]) sonarResBody _: = ioutil.ReadAll (sonarRes.Body) var sonarJson map [string] interface {} var sonarStatus stringjson.Unmarshal (sonarResBody, & sonarJson) if sonarJson ["task"] ["warningCount"]. (float64) = 0 {sonarStatus = ": white_check_mark:" d = "No Warings"} else {sonarStatus = ": X:" d = fmt.Sprintf ("Warings:% f") SonarJson ["task"] ["warningCount"]. (float64)} / / othersif args [4] = = "invalid" {a = "Please build locally before submitting"} if args [5] = "invalid" {b = "duplicate submission"} if args [6] = "invalid" {c = "incorrect syncWhitelist format" Please confirm "} var botMsg stringif a = =" success "& & b = =" success "& & c = =" success "{botMsg =" all checks passed, please @ Zoker process "} else {botMsg =" detected * * failed items * *, please check the adjustment and resubmit, the check will be triggered automatically. " } params: = make (map [string] interface {}) params ["access_token"] = TOKEN params ["body"] = fmt.Sprintf (`Automated detection completed Please adjust according to the result: | step | result | remarks | |-|-- |-| build |% s |% s | | submit information check |% s |% s | | syncWhitelist format check |% s |% S | | Sonar quality test |% s |% s |% s` Status [args [4]], a, status [args [5]], b, status [args [6]], c, sonarStatus, d, botMsg) commentUrl: = fmt.Sprintf ("https://gitee.com/api/v5/repos/%s/%s/pulls/%s/comments", OWNER, REPO, args [2]) postForm (commentUrl, params)}

In the end, our build script looks like this:

# add welcome msg when PR createdstatus= `/ home/zoker/app/jenkins/scripts/giteeCheck welcome ${giteePullRequestIid} `if [$status= = "1]; then # check syncWhitefile validation syncFormat=` / home/zoker/app/jenkins/scripts/giteeCheck checkfile config/ syncWhitelist` # check commit msg commitCountA= `git log HEAD ^ remotes/origin/master-- pretty=format:'%B' | sort-r | grep-v'^ $'| wc-l`commitCountB= `git log HEAD ^ remotes/origin/master-- pretty=format:'%B' | sort-r | grep-v'^ $'| uniq | wc-l`if [$commitCountA= = $commitCountB] Then commitMsg= "valid" else commitMsg= "invalid" fi # build and check format go build if [- x gau]; then buildStatus= "valid" else buildStatus= "invalid" fi # Sonar check / home/zoker/app/sonar/scanner/bin/sonar-scanner-Dsonar.projectKey=GAU-Dsonar.sources=. -Dsonar.host.url= http://127.0.0.1:8089-Dsonar.login=410exxxxxxxxxxxxxxxxxxx3faa9430f7c6 > sonarRes sonarResUrl= `cat sonarRes | grep api/ce | cut-d''- f8`cat 3 # callback result / home/zoker/app/jenkins/scripts/giteeCheck result ${giteePullRequestIid} ${sonarResUrl} $buildStatus $commitMsg $syncFormatfi verification effect

Let's contribute the code to fix a forced push problem for GAU. Let's create a new fix-force-push branch and modify the code on this branch:

Refine the submission information update utils.go fixed force push bug and submit it to the branch, then we create a PullRequest

When we enter the Jenkins, we can see that the build has been triggered:

And as we expected, there was a welcome message.

We comment that Rebuild accepts the Contributor Agreement and starts checking, waiting for a moment to see the result:

As you can see, the build failed because the Jenkins build machine did not configure the agent, and the download of the dependency package timed out. After processing, we re-executed Rebuild.

As you can see, due to the small amount of code, there are no quality problems with Sonar. What needs to be noted here is that in general, updating the code will automatically trigger the build, so there is no need for us to comment on Rebuild manually.

This is the end of the "Gitee + Jenkins-based Open Source Project Automation collaboration method tutorial". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.