Shulou(Shulou.com)06/02 Report--

What is CI/CD?

The terms continuous integration Continuous Integration (CI) and continuous delivery Continuous Delivery (CD) are often mentioned in software development. But what do they really mean?

In this article, I will explain the meaning and meaning behind these and related terms, such as continuous testing Continuous Testing and continuous deployment of Continuous Deployment.

The assembly line in the factory produces consumer goods from raw materials in a fast, automated and repeatable manner. Similarly, the software delivery pipeline generates releases from source code in a fast, automated, and repeatable manner. The overall design of how to do this is called "continuous delivery" (CD). The process of starting an assembly line is called continuous integration (CI). The process of ensuring quality is called "continuous testing", and the process of providing the final product to users is called "continuous deployment". Some experts who make all this simple, smooth, and efficient are called DevOps practitioners of operation and maintenance development.

"continuous" is used to describe following many of the different process practices I have mentioned here. This does not mean "running all the time", but "running at any time". In the area of software development, it also includes several core concepts / best practices. These are:

Frequent releases: the goal behind continuous practice is to be able to deliver high-quality software frequently. The delivery frequency here is variable and can be defined by the development team or company. For some products, once a quarter, a month, a week, or a day may be frequent enough. For others, it is possible that multiple deliveries may be required in a day. The so-called continuity also has "occasional, on-demand" aspects. The ultimate goal is the same: to provide high-quality software updates to end users in a repeatable and reliable process. Typically, this can be done with little or no user interaction or knowledge (think about device updates). Automated process: the key to achieving this frequency is to use automated processes to deal with all aspects of software production. This includes building, testing, analyzing, versioning, and in some cases deployment. Repeatable: if the automated process we use always has the same behavior given the same input, the process should be repeatable. That is, if we take a historical version of the code as input, we should get the same deliverable output. This also assumes that we have external dependencies of the same version (that is, we do not create other deliverables used by this version of the code). Ideally, this also means that the process in the pipeline can be versioned and rebuilt (see the DevOps discussion later). Fast iteration: "fast" is a relative term here, but regardless of the frequency of software updates / releases, the expected continuous process will convert source code into deliverables in an efficient manner. Automation is responsible for most of the work, but the process of automation may still be slow. For example, for products that need to release candidate updates multiple times a day, it may be too slow for a round of integration testing integrated testing to take more than half a day.

Several different tasks task and job job that convert the source code into a releasable product are usually concatenated into a software "pipeline", and an automated process starts the next process in the pipeline after successful completion. These pipes have many different names, such as continuous delivery pipeline, deployment pipeline, and software development pipeline. In general, program managers manage the definition, operation, monitoring, and reporting of various parts of the pipeline during pipeline execution.

The actual implementation of the software delivery pipeline can be very different. There are many programs available in pipelines for source code tracking, building, testing, metrics collection, version management, and so on. But the overall workflow is usually the same. A single business process / workflow application manages the entire pipeline, and each process runs as a separate job or is managed in phases by the application. Typically, in a business process, these stand-alone jobs are defined by the syntax and structure that the application can understand and can be used as workflow management.

These jobs are used for one or more functions (build, test, deploy, etc.). Each job may use a different technology or multiple technologies. The key is that the job is automated, efficient, and repeatable. If the job is successful, the Workflow Manager triggers the next job in the pipe. If the job fails, the workflow manager alerts developers, testers, and others so that they can correct the problem as soon as possible. This process is automated, so errors can be found faster than running a set of processes manually. This fast troubleshooting is called a quick failure fail fast and is also valuable in reaching the end of the pipe.

One of the jobs of pipelines is to deal with changes quickly. The other is to monitor the different tasks / jobs that create the publication. Because failed compilation or failed tests can prevent the pipeline from continuing to run, it is important to quickly notify the user of this situation. Quick failure refers to the way to find the problem in the pipeline process as soon as possible and notify the user quickly, so that the problem can be corrected in time and the code can be resubmitted to make the pipeline run again. Usually in the pipeline process, you can look at the history to determine who made the change and notify that person and his team.

Almost all parts of the pipeline should be automated. For some parts, there are places where human intervention / interaction may make sense. One example might be the user acceptance test user-acceptance testing (let end users try the software and make sure it reaches the level they want / expect). Another scenario may be that users want more human control when deployed to a production environment. Of course, if the code is incorrect or does not work, human intervention is required.

With the background of understanding the meaning of "continuous", let's look at the different types of ongoing processes and their meaning in the context of software pipelines.

Continuous integration (CI) is the process of automatically detecting, pulling, building, and (in most cases) unit testing after source code changes. Continuous integration is the part of the pipeline that starts (although some pre-validations-- often referred to as pre-launch pre-flight checks-- are sometimes classified before continuous integration).

The goal of continuous integration is to quickly ensure that newly submitted changes by developers are good and suitable for further use in the code base.

The basic idea of continuous integration is to have an automated process monitor whether one or more source code repositories have changed. When changes are pushed to the warehouse, it detects changes, downloads copies, builds, and runs any relevant unit tests.

Currently, the monitor is usually an application like Jenkins, and it also coordinates all (or most) processes running in the pipeline, and monitoring for changes is one of its functions. Monitoring procedures can monitor changes in several different ways. These include:

Polling: the monitor repeatedly asks the code management system, "is there anything new I'm interested in in the code repository?" When there are new changes to the code management system, the monitor "wakes up" and completes its work to get the new code and build / test it. Periodic: the monitor is configured to start the build periodically, regardless of whether the source code has changed or not. Ideally, if there are no changes, no new content will be built, so there is no additional cost. Push: this is the opposite of the monitor used for code management system checks. In this case, the code management system is configured to "push" a notification to the monitor when the change is submitted to the warehouse. Most often, this can be done in the form of webhook-a program that ticks hook sends a notification to the monitor over the Internet when the new code is pushed. To do this, the monitoring program must have an open port that can receive webhook information over the network.

Additional validation can be done before introducing the code into the repository and triggering continuous integration. This follows best practices, such as test build test build and code review code review. They are usually built into the development process before the code is introduced into the pipeline. But some channels may also use them as part of their monitoring process or workflow.

For example, a tool called Gerrit allows formal code review, validation, and test builds after developers push code but before allowing access to (Git remote) repositories. Gerrit is located between the developer's workspace and the Git remote repository. It "receives" pushes from developers and can perform pass / fail validation to ensure that their checks pass before they are allowed into the warehouse. This can include detecting new changes and starting build tests (a form of CI). It also allows developers to conduct formal code reviews at that time. This approach has an additional credibility assessment mechanism that does not break anything when the changed code is merged into the code base.

Unit tests (also known as "commit tests") are small, specialized tests written by developers to ensure that new code works independently. "independent" here means not relying on or calling other code that is not directly accessible, nor on external data sources or other modules. If such dependencies are required to run the code, these resources can be represented as a mock mock. Impersonation refers to the use of a code stub code stub that looks like a resource and can return a value without implementing any functionality.

In most organizations, developers are responsible for creating unit tests to prove that their code is correct. In fact, a model called test-driven development test-driven develop (TDD) requires the first design of unit tests as the basis for clearly verifying the functionality of the code. Because such code can be changed quickly and with a lot of change, they must also be executed quickly.

Because this is related to continuous integration workflows, developers write or update code in the local work environment and pass unit tests to ensure that the newly developed features or methods are correct. Typically, these tests take the form of assertions, that is, a given input set of a function or method produces a given output set. They are usually tested to ensure that error conditions are properly marked and handled. There are many unit testing frameworks that are useful, such as JUnit for Java development.

Continuous testing is the practice of running extended-scope automated tests when code passes through the continuous delivery pipeline. Unit testing is usually integrated with the build process as part of the continuous integration phase and focuses on testing isolated from other code that interacts with it.

In addition, there can or should be various forms of testing. These may include:

Integration testing verifies that components and services are properly combined. Functional testing verifies that the results of performing functions in the product are as expected. Acceptance tests validate some characteristics of the product according to acceptable standards. Such as performance, scalability, compression capacity and capacity.

All of this may not exist in the automated pipeline, and the classification boundaries of some different types of tests are not very clear. However, the goal of continuous testing in the delivery pipeline is always the same: to prove that the quality of the code can be used in an ongoing release through a continuous test level. Based on the principle of continuous integration speed, the second goal is to quickly identify problems and alert the development team. This is often referred to as quick failure.

In addition to whether the test passes or not, there are some applications that can tell us the number of lines of source code that the test case executes (covers). This is an example of how you can measure the amount of code. This metric is called code coverage code-coverage and can be counted by tools such as JaCoCo for Java.

There are many other types of metric statistics, such as the number of lines of code, complexity and comparative analysis of code structure. Tools such as SonarQube can examine the source code and calculate these metrics. In addition, users can set thresholds for the "qualified" range of indicators that they can accept. You can then set a check for these thresholds in the pipeline, on the process terminal if the result is not within an acceptable range. Applications such as SonarQube are highly configurable and can be set to check only what the team is interested in.

Continuous delivery (CD) usually refers to the entire process chain (pipeline), which automatically monitors source code changes and runs them through build, test, packaging, and related operations to produce deployable versions, with virtually no human intervention.

The goal of continuous delivery in the software development process is automation, efficiency, reliability, repeatability and quality assurance (through continuous testing).

Continuous delivery includes continuous integration (automatically detecting source code changes, performing the build process, running unit tests to verify changes), continuous testing (running various tests on code to ensure code quality), and (optional) continuous deployment (automatically available to users through a pipeline release version).

Version control is a key concept of continuous delivery and pipelining. Persistence means being able to integrate new code and provide updated versions on a regular basis. But that doesn't mean everyone wants the "latest and best". This is especially true for internal teams that want to develop or test a known stable version. Therefore, it is important for pipes to create and easily store and access these versioned objects.

Objects created from source code in pipes can often be called artifacts artifact. Artifacts should have versions applied to them when they are built. The recommendation strategy for assigning version numbers to artifacts is called semantic version control semantic versioning. (this also applies to versions of dependent artifacts introduced from external sources. )

The semantic version number has three parts: the major version major, the minor version minor, and the patch version patch. (for example, 1.4.3 reflects major version 1, minor version 4, and patch version 3. The idea is that one part of the change represents the update level in the artifact The major version is incremented only for incompatible API changes. Minor versions are added when features are added in a way that is backward compatible with backward-compatible. When a backward compatible version of bug is fixed, the patch version is added. These are recommended guidelines, but as long as the team does so in a consistent and easy-to-understand manner throughout the organization, the team is free to change this approach. For example, the number added each time the build is completed for the release can be placed in the patch field.

The team can assign a distribution promotion level to the artifact to indicate that it is suitable for testing, production, and other environments or uses. There are many ways. You can use applications such as Jenkins or Artifactory for distribution. Or a simple solution can add a tag to the end of the version number string. For example,-snapshot can indicate the latest version (snapshot) of the code used to build the artifact. Various distribution strategies or tools can be used to "promote" artifacts to other levels, such as-milestone or-production, as markers for stability and completeness versions of artifacts.

Versioned artifacts built from source code can be stored through an application that manages the artifact repository artifact repository. An artifact repository is like a version control tool for building artifacts. Applications like Artifactory or Nexus can accept versioned artifacts, store and track them, and provide methods for retrieval.

Pipe users can specify the versions they want to use and use pipes in those versions.

Continuous deployment (CD) is the idea of automatically providing release versions in the continuous delivery pipeline for end users to use. Depending on how the user is installed, it may be automatic deployment in the cloud, app upgrade (such as applications on mobile phones), update the website, or only update the list of available versions.

A key point here is that just because it can be continuously deployed does not mean that every set of deliverables from the pipeline is always deployed. It actually means that every set of deliverables proved to be "deployable" through pipelines. This is largely done by the continuous level of continuous testing (see the continuous testing section in this article).

The deployment of pipeline-built releases can be controlled by manual decisions or by using various methods of "trial" release before full deployment.

Because having to roll back / undeploy all users can be a costly situation (both technically and in the user's perception), there are already many technologies that allow you to "try" to deploy new features and easily "undo" them when problems are found. These include

In this method of deploying software, two identical host environments are maintained-one "blue" and one "green". (color is not important, it is only used as an identification. One of them is a "production environment" and the other is a "pre-release environment".

In front of these examples are scheduling systems, which act as customer "gateways" for products or applications. By pointing the scheduling system to a blue or green instance, customer traffic can be directed to the desired deployment environment. In this way, switching to which deployment instance (blue or green) is fast, simple, and transparent for the user.

When the new version is ready for testing, it can be deployed to a non-production environment. After testing and approval, you can change the scheduling system settings to direct incoming online traffic to it (so it will become the new production site). Now, it has been used as an example of a production environment for the next candidate release.

Similarly, if a problem is found in the latest deployment and the previous production instance is still available, simple changes can divert customer traffic back to the previous production instance-effectively "offline" the problem instance and rolling back to the previous version. New instances with problems can then be fixed in other areas.

In some cases, switching the entire deployment through a blue / green release may not be feasible or expected. Another approach is to test / deploy for Canary canary. In this model, some customer traffic is redirected to the new version deployment. For example, a new version of the search service can be deployed with the production version of the current service. 10% of the search queries can then be drained to the new version to test them in a production environment.

If there is no problem with the new version of serving that traffic, more traffic may be gradually diverted. If there are still no problems, you can deploy the new version incrementally over time until 100% of the traffic is scheduled to the new version. This effectively "replaces" the previous version of the service and makes the new version effective for all customers.

For new features that may need to be easily turned off (if problems are found), developers can add the feature switch feature toggles. This is the if-then software feature switch in the code, and the new code is activated only when the data value is set. This data value can be a globally accessible location, and the deployed application will check whether the new code should be executed at that location. If the data value is set, the code is executed; if not, it is not executed.

This provides developers with a remote "termination switch" to turn off new features when problems are found after deployment to a production environment.

In the clandestine release dark launch, the code is gradually tested / deployed to the production environment, but the user does not see the change (hence the word black-box dark in the name). For example, in the production version, some parts of a Web page query may be redirected to a service that queries a new data source. Developers can collect this information for analysis without exposing any information about interfaces, transactions, or results to users.

The idea is to get real information about how the candidate version performs under the load of the production environment without affecting users or changing their experience. Over time, more load can be scheduled until a problem is encountered or the new feature is thought to be ready for everyone to use. In fact, the function switch flag can be used in this dark box publishing mechanism.

DevOps is a series of ideas and recommended practices on how to make it easier for development and operations teams to work together to develop and distribute software. Historically, the development team has developed products, but did not install / deploy them in a conventional, repeatable manner as customers do. Throughout the cycle, this set of installation / deployment tasks (and other support tasks) are left to the operations team. This often leads to a lot of confusion and problems because the operation and maintenance team begins to intervene at a later stage and must finish their work in a short period of time. Similarly, development teams are often at a disadvantage-they may be surprised by the problems that arise in the process because they do not adequately test the installation / deployment capabilities of the product.

This often leads to serious disconnection and lack of cooperation between the development and operation and maintenance teams. The DevOps concept advocates an integrated collaboration between development and operation and maintenance throughout the development cycle, just like continuous delivery.

The continuous delivery pipeline is an implementation of several DevOps concepts. Later stages of product development, such as packaging and deployment, can always be completed in each run of the pipeline, rather than waiting for a specific time in the product development cycle. Similarly, from development to deployment, developers and operators can clearly see when things work and when they don't. To make the continuous delivery pipeline cycle successful, it is necessary to go through not only the development-related processes, but also the operations-related processes.

To go further, DevOps suggests that the infrastructure that implements the pipeline will also be treated as code. That is, it should be automatically configured, traceable, easy to modify, and trigger a new round of operation when the pipeline changes. This can be done by implementing the pipeline as code.

Pipe or code pipeline-as-code is a generic term for creating pipeline jobs / tasks by writing code, just like developers write code. Its goal is to represent the pipeline implementation as code so that it can be stored, reviewed, tracked with the code, and can be easily rebuilt if there is a problem and the pipe must be terminated. There are several tools that allow this, such as Jenkins 2.

In the traditional sense, each hardware system used in the pipeline has supporting software (operating system, application, development tools, etc.). In extreme cases, each system is set up manually to customize. This means that this is usually a custom task when there is a problem with the system or when an update is needed. This approach runs counter to the basic concept of continuous delivery, which is an environment that is easy to reproduce and traceable.

Over the years, many applications have been developed for standardized delivery (installation and configuration) systems. Similarly, the virtual machine virtual machine is developed to simulate computer programs that run on other computers. These VM require hypervisors to run on the underlying host system, and they need their own copy of the operating system to run.

Then came the container container. Containers are similar in concept to VM, but work differently. They only need to use some existing operating system structures to divide the isolation space, rather than running separate programs and copies of the operating system. Therefore, they behave like VM to provide isolation but do not require too much overhead.

VM and containers are created based on configuration definitions, so they can be easily destroyed and rebuilt without affecting the host system on which they are running. This allows the system running the pipe to be rebuilt. In addition, for containers, we can track changes to its build definition file-- just like for source code.

Therefore, if we encounter problems in VM or containers, we can destroy and rebuild them more easily and more quickly, rather than trying to debug and fix them in the current environment.

This also means that any change to the pipeline code can trigger a new run of the pipeline (through CI), just like a change to the code. This is one of DevOps's core ideas about infrastructure.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.