Shulou(Shulou.com)06/03 Report--

Static code scanning adds value to the entire development organization. Regardless of your role in the development organization, static code scanning solutions have added value, with the cutting-edge features needed in software development to maximize quality and manage risks in software products.

Background

The micro-service architecture model has the characteristics of independent service, independent development and deployment, independent development induces technical separation, and HTTP communication increases the complexity of problem diagnosis, which brings great challenges to the quality assurance of system function, performance and security.

"

The Challenge of Microservice Architecture to testing

Under the micro-service architecture model, multiple independent business services are developed at the same time, and each system has its own business scope and development cycle requirements. As a result, in the traditional process shown in the figure below, the product manager provides the requirements. The requirements are analyzed by the requirements personnel, developed by the developers, and finally tested by the testers, which cannot meet the requirements of test coverage and test efficiency.

Compared with the traditional monolithic model, the challenges to testing under the microservice model can be summarized as follows:

1. As the modules of the micro-service system are hierarchical, it is necessary to ensure the quality of the internal code of the module. In this scenario, the traditional end-to-end testing can not meet the testing requirements.

two。 It is necessary to ensure the correctness of the internal modules of each micro-service system. Development work is usually carried out between system modules, front-end and back-end at the same time. Modules or front and back ends are connected through interfaces (usually Restful http interfaces), while modules and back-ends often have no interface. In order to ensure the correctness of individual dependent systems of each system, it is necessary to test at the interface level with the help of Mock technology to isolate dependencies.

3. It is necessary to ensure the consistency of the interface in the microservice system, that is, the consistency of the contract. It is necessary to ensure the correctness of the contract by means of contract testing, so as to ensure the correctness and consistency of pre-and post-development in the process of synchronous development.

4. It is necessary to ensure the correctness of a single microservice system. The correctness of the microservice system needs to be tested at the component level.

5. The correctness of the whole system needs to be guaranteed. After each micro-service system is connected in series, the correctness of the whole system is ensured by end-to-end testing.

"

How to carry out testing under micro-service architecture

In view of the testing challenges of microservices mentioned above, on the one hand, in order to ensure comprehensive testing of microservices at all levels of services, especially for distributed systems, and on the other hand, to ensure the efficiency of test execution, so as to ensure continuous integration / continuous delivery (CI/CD). Therefore, the overall testing strategy adopts the following solutions:

1. Develop the "quality" culture. Let developers establish the awareness of code "quality" to ensure the quality within the module.

two。 The automatic testing method is adopted. In the micro-service architecture, development is divided into multiple groups responsible for different services, and testers often spend a lot of time every day to understand the development progress of different teams. If you still need to do regression testing (Regression Test) manually, you will eventually be overwhelmed. Therefore, automated testing is a necessary means in the micro-service mode.

3. Hierarchical automated testing strategy. Automated test layering is described in detail in the principle of test pyramid (Test Pyramid) proposed by Mike Cohn. It advocates different granularity testing at code level, interface level and application level to ensure the quality of the system. From the perspective of the proportion of investment in automated testing, unit testing and static code scanning have the largest proportion of investment, followed by interface automated testing, and finally UI automated testing. At the same time, in order to improve test efficiency and test coverage, functional testing needs the help of exploratory testing means to carry out testing.

4. The pipeline technology is used for visual and fast feedback. Because there are so many micro-service systems, it often increases the cost of operation, maintenance and communication. In order to improve the efficiency of communication, it is necessary to visually view each Build, Test and Deploy process with the help of pipeline technology, and make quality feedback and processing decisions quickly. Through the visual pipeline, we can finally realize the monitoring of each link, and use DevOps means to break through the departmental walls of business, development, testing and operation and maintenance.

Combined with the idea of hierarchical automated testing, static code scanning is introduced first.

Static code scan

"

Static code scans the background

Static code analysis refers to the technology of scanning program code through lexical analysis, syntax analysis, control flow, data flow analysis and so on without running the code. Its purpose is to verify whether the code meets the requirements of standardization, security, reliability and maintainability. Static code scanning is at the bottom of hierarchical automated testing, at the same level as unit testing. In order to ensure the standardization, security and reliability of the company code, the company-level static code scanning specifications, scanning rules and scanning implementation process are customized to ensure efficient landing.

"

Significance of static code scanning

For developers

Software developers are ultimately responsible for code quality. Code quality is part of the non-functional requirements and is therefore the direct responsibility of the developer. There should be no technical debt for code quality, and feedback should be provided at every step of the development process, from IDE to release. This enables developers to make decisions about code quality as soon as possible, so that they can do better and provide better quality software products.

For DevOps

DevOps needs to ensure that the software is built correctly. There are many responsibilities involved in DevOps, including supporting development processes, automating testing, ensuring quality, and improving productivity. And finally achieve continuous deployment. Good code quality is a necessary, though not sufficient, condition for achieving all these goals. Static code scanning can add code quality inspection thresholds to any build / test / deployment step and automatically enforce a uniform set of quality standards to ensure that organizations deliver better software.

For the manager

Static code scanning reduces risk and increases team productivity. Managers need to be able to run the software safely and spend a reasonable return on investment. Our solution clearly shows the technical debt they face and the cost of their mitigation. It also has out-of-the-box functionality to systematically improve the maintainability and long-term productivity of the development team. This enables managers to use risk control methods at the best cost to ensure that their organizations are able to deliver better software.

"

Introduction to static code scanning

Static code scanning is after the development of the feature branch is completed, which is described as follows:

1. Developers pull the feature branch from the Master branch as the development branch

two。 After developing the feature branch, code building, unit testing, static code scanning

3. After passing, it is merged into the Master branch for production.

"

Static code scanning process

Pay-as-you-go static code scanning platform is realized by integrating SonarQube platform tools, Jenkins integration tools, IDE SonarLint plug-ins and CheckStyle localization rule templates and other open source tools and plug-ins. To achieve the implementation of localization code detection, version construction after the secondary detection, as well as mail feedback and other functions of the process closed loop to ensure that the pre-production code in line with the requirements of the pay code specification. The specific process is shown in the following figure:

1. Localized IDE implements synchronization with SonarQube platform rules and specifications through SonarLint plug-ins, and realizes local code checking; along with customized java rules, XML rules 257, javascript rules 86, used to detect code standardization, code defects, vulnerabilities, bad taste, repetition rate and other information. And put the customized rules on the SonarQube platform, the rules of the SonrLint plug-in are more comprehensive, including all sonajava rules and javascript rules. In order to ensure the local use of customized rules, and consistent with the rules in sonarqube, you need to remotely connect to the SonarQube server and bind the project. Take Eclipse as an example to show the process of SonarQube connection and project binding:

two。 After the code is submitted to the code base GitLab, when the test environment is built in Jenkins, the Sonar scan is automatically triggered and the scan results are published to the SonarQube platform. The following figure shows the results of a project shown by SonarQube:

3.SonarQube platform according to the quality valve requirements, do not meet the quality valve requirements, then notify the developer by email. Quality valve requirements: 1. The new coverage is greater than or equal to 80%. The new Bugs is 0; 3. The new loophole is 0; 4. Added bad taste to 0

4. After receiving the email, the developer processes the code until the specification requirements are met.

5. Count the number of Bugs\ loopholes\ bad smell scanned by static code in SonarQube platform on a weekly basis, and automatically send weekly reports to relevant stakeholders on a regular basis, and the report will include a trend chart of problem handling.

SonarQube and rules

SonarQube is an open source platform for code quality management, supporting quality scanning in 25 + programming languages. SonqrQube is composed of remote computer, server and database. The remote client can upload the analyzed project code to SonarQube server and manage and analyze the code quality through a variety of analysis mechanisms. SonarQube will also display the analysis results in a visual and measurable way through Web API. The logical structure is shown in the following figure:

"

Integration capabilities of SonarQube

The integration of various static code scan detection tools is supported in the SonarQube platform. Comparison of various code detection tools in SonarQube to analyze objects and application technologies:

Java static analysis tool analysis object application technology CheckStyleJava source file defect pattern matching FindBugs bytecode defect pattern matching; data flow analysis PMDJava source code defect pattern matching CheckStyle

It can easily help us check for format errors in Java code, and it can automate the code specification checking process, thus freeing developers from this important but boring task. Basically, the rules are customized according to the development rules. It mainly covers the following contents:

Javadoc comments: check Javadoc comments for classes and methods

Naming conventions: check whether naming conforms to naming conventions

Title: check whether the file begins with certain lines

Import statement: check whether the Import statement conforms to the definition specification

Code block size, that is, the number of lines of code blocks such as classes, methods, etc.

Whitespace: check for blank characters, such as tab, carriage return, etc.

A check of decorating symbols, such as the order in which modifiers are defined.

Blocks: check for empty or invalid blocks

Code problems: check duplicate code, condition judgment, magic number, etc.

Class design: check whether the definition of the class conforms to the specification, such as the definition of the constructor FindBugs

Findbugs is a static analysis tool that examines classes or JAR files and compares bytecode to a set of defect patterns to find possible problems. It mainly covers the following contents:

Bad practice bad practice: common code errors for defect pattern matching during static code review

Correctness may cause incorrect code, such as null pointer references, etc.

Problems related to internationalization, such as incorrect string conversion

Possible malicious * *, such as the definition of access modifier, etc.

The correctness of multithreading: such as synchronization and thread scheduling problems in multithreaded programming.

Runtime performance issues: code inefficiency problems caused by variable definitions and method calls PMD

An open source tool for analyzing errors in Java code based on the use of JavaCC generation parsers to parse the source code and generate AST (abstract syntax trees). Unlike other analysis tools, PMD learns about code errors through static analysis. That is, errors are reported without running the Java program. PMD comes with many rules that can be used directly, which can be used to identify many problems with Java source programs, such as:

Potential Bugs: check for potential code errors, such as empty try/catch/finally/switch statements

Unused code (Dead code): check for unused variables, parameters, methods, etc.

Optional code: abuse of String/StringBuffer

Complex expressions: check for unnecessary if statements, for loops that can be replaced by while

Duplicate code: check for duplicate code

Loop body create new object: check that the new object is instantiated in the loop body

Resource shutdown: check whether resources such as Connect,Result,Statement are closed after use

In addition, users can define their own rules to check whether the Java code conforms to certain coding specifications. For example, you can write a rule that requires PMD to find out all the actions that create Thread and Socket objects.

Comparison of three tools

You can see from the table that several tools have their own emphasis on code review. Among them, Checkstyle pays more attention to the format of code writing and the test of whether it conforms to the coding specification, and the discovery function of code bug is weak, while FindBugs,PMD focuses on finding code defects. In the code defect inspection, the three tools are different in the code defect categories, and there is overlap between the categories.

"

Rule customization

Considering that the Sonar Java rules already contain PMD and CheckStyle rules, we chose the default rule of Sonar and customized it. Part of the customized rule content is shown in SonarQube in the following figure.

The types of code defects covered by the customized rules are shown in the following table (some rules):

Code defect classification example reference operation Null pointer reference object Operand comparison (use = = instead of equals) expression complicates an array of if statements using an array subscript out of bounds unused variables or code snippets unused variable resource recovery unclosed method call unused method return value code to design an empty try/catch/finally block

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.