Shulou(Shulou.com)06/01 Report--

This article is about how infrastructure codification IaC can be configured and orchestrated automatically. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

Cloud operation and maintenance, that is, dealing with cloud resources and products, will undoubtedly involve a series of resource deployment. For example, simply using a CVM requires OPS personnel to create VPC, VSwitch, security groups and CVM instances in turn. If you want to create a cluster, you need to further create load balancers, databases and multiple CVM instances.

With the continuous expansion of the business scale and the increasing complexity of the IT system and environment, it is obviously not advisable to create resources one by one, and many people are turning to tools to automate the deployment and configuration of resources.

Based on the concept of infrastructure, that is, code IaC, this article will share how to use automated orchestration tools to achieve automated deployment, so as to make the work of operation and maintenance more efficient.

Five pain points of manual / semi-manual cloud operation and maintenance

For the deployment of cloud resources, if your cloud OPS is still in the manual or semi-manual OPS stage, most of the work is to select specific resource specification parameters through the console to create resources, and some are to use CLI (such as aliyun-cli) or SDK to directly call the API to create resources. However, as the scale of an enterprise's cloud business continues to expand, either way, it will more or less encounter the following five problems:

The deployment efficiency is low. Manual creation is a very intuitive way to create a small number of kinds of resources, but when a large number of different resources are involved, especially when there are dependencies between resources, you will find that you need to switch back and forth between different product consoles. you have to keep an eye on the creation progress before you can create the next resource that depends on it, and you can imagine the time and effort spent in the whole process. I believe many people have deep experience.

But the replication is poor. After manually creating a series of resources, if you need to create identical resources for different environments (such as pre-delivery, testing and production) or different regions (such as Beijing and Shanghai), then it takes a lot of time to operate step by step, unable to directly copy and achieve one-click deployment.

Poor consistency. There is also a very big problem with manual creation, that is, configuration errors are very easy to occur, and it is difficult to ensure that the resource configurations in the two environments are exactly the same.

It is difficult to manage. The creation of resources is just the beginning, and you may need to scale up and update the specifications of specific resources for these resources. However, the manual operation and maintenance method leads to no unified management of these resources, so it still needs to be operated separately on each product console. With the increasing number of resources, resource management becomes more and more difficult to maintain.

Difficult to DevOps. Each time a software application is developed, tested, or deployed, the infrastructure may need to be deployed manually, with neither version control nor review of changes to the infrastructure, nor agile deployment.

In fact, we all know that the reason behind these problems is that the deployment of resources is not yet automated. But these problems also constantly prompt us to think about how to solve these pain points in order to automate the entire resource deployment process.

Introduce the concept of infrastructure, namely code IaC, to realize automatic deployment of resources on the cloud.

Before you really automate the deployment, you might as well look back at the cloud service resources you need to create (such as VPC, VSwitch, ECS instances, etc.), which are all cloud infrastructure relative to applications such as Web services. If you think of these infrastructure as a piece of "code" and define product, specification, quantity and other information in the "code". So is it possible to manage the entire infrastructure through this "code"?

This is the concept of infrastructure as Infrastructure as Code (IaC), which treats infrastructure configuration as software programming. This is how Kief Morris defines infrastructure as code in his book Infarftruce as Code:

"Infrastructure is code is a way to use new technologies to build and manage dynamic infrastructure. it treats infrastructure, tools, and services, and the management of infrastructure itself as a software system, and adopts software engineering practices to manage changes to the system in a structured and secure manner."

By introducing the concept of IaC, operators can make the deployment and management of infrastructure agile:

Define infrastructure in templates (code in a broad sense), that is, attributes such as various cloud resources and their specifications, quantities, and dependencies between cloud resources.

Use version control (such as Git) to manage templates and submit for review

After review, automated deployment tools use templates to create / update infrastructure

After the deployment and management of infrastructure becomes convenient, the pain points of manual OPS / semi-manual OPS mentioned above can be well resolved:

Improve deployment efficiency. Using automated deployment tools for deployment will be much more efficient than manual deployment.

Standardization and consistency. The content of the infrastructure is saved in the form of a template, and the changes to the infrastructure are realized by the changes to the template, thus realizing the standardization of infrastructure management. In addition, using the same template to deploy in different regions can also ensure the consistency of resources.

Easy to manage. The management of infrastructure is no longer dispersed in each product console, but unified into a single template, which greatly reduces the management cost.

Agile workflow. Through the standardization and standardization of infrastructure management processes, the entire process of resource deployment becomes agile.

Audit and rollback. Versioning of templates makes it possible to audit and roll back infrastructure changes to a specific version.

Four common IaC automatic configuration and orchestration tools

Currently, there are many IaC automation deployment tools, including third-party resource orchestration tools, and cloud native resource orchestration tools provided by cloud service providers. Here are four automated configuration and orchestration tools:

Ali Cloud resource orchestration service ROS (Resource Orchestration Service), which is a cloud native orchestration tool. By writing a template in JSON/YAML format, the required cloud service resources such as ECS instance and database instance, as well as resource dependencies are defined in the template, and then the resource stack is created in ROS according to the template. The ROS server will automatically complete the creation and configuration of all resources according to the template, realizing automatic deployment and operation and maintenance. On the other hand, the resource stack manages all the resources defined in the template, and can update the resource stack through the new template, including the operation of adding, updating or deleting resources.

AWS CloudFormation is also a native cloud orchestration tool. Operators also define cloud service resources through templates in JSON/YAML format and manage these resources through the resource stack.

HashiCorp Terraform, an open source automated orchestration tool. Driven by the configuration file, you can define the components to be managed, that is, infrastructure resources, in the file, so as to generate an executable plan, which can be executed to complete the creation of the defined components, incremental changes and continuous management. If it cannot be executed, it will prompt for an error. Terraform can not only manage IaaS layer resources, such as computing instances, network instances and storage instances, but also manage higher-level services, such as DNS domain name and resolution records, SaaS application functions and so on.

Pulumi, like Terraform, is an open source project, but an important difference from Terraform is that declarative configurations can be written in familiar programming languages without the need to learn additional cloud provider-specific template languages to write configurations.

For the choice of automatic configuration and orchestration tools, the author's suggestions are:

If your business is deployed on a single cloud platform, choose the resource orchestration tool provided by the cloud platform. Use ROS on the Aliyun platform and CloudFormation on the AWS platform. The reason is simple: the tools provided by the cloud platform are cloud native, free managed services, and can be deployed automatically on the server side. At the same time, it also realizes the functions of cloud native access control, choreography resources and actual resource difference detection, which is easy to use.

If your business is deployed on multiple cloud platforms, it is recommended to use third-party Terraform and Pulumi, because it can not only deploy and manage multi-cloud resources, but also manage other resources other than cloud, such as Kubernetes.

How to use orchestration tools for automated deployment and management?

For operators, the threshold for automated deployment tools using the concept of IaC is not high, and the steps to use are very simple, mainly writing templates and using templates. Here we talk about the precautions of writing and using templates, and how to make better use of the tools and improve the efficiency of operation and maintenance.

1. Three points for attention in writing a template

After confirming the automated deployment tools, you can write the corresponding template files according to the template languages of different tools. If you choose the cloud native orchestration tool provided by the cloud service provider, write a template here, there are three points to remind you:

Pay attention to the dependency of resources. Inappropriate dependencies or lack of dependencies can lead to resource creation errors.

Note that generic properties are used as parameters. For example, instance specifications are common attributes. It is recommended to use the same template and specify different parameters to achieve the purpose of deploying instances with different specifications.

Use valuable attributes as output. For example, instance ID, connection address and other contents are valuable attributes, which can only be obtained after the resource has been created. Using these attributes as the output of the whole template can be easily viewed and managed later.

2. Automatically resolve dependencies and deploy infrastructure automatically

After you have written the template, you can convert the template into a real resource through the corresponding automated deployment tool. All of the orchestration tools mentioned above can resolve the dependencies of resources and create them one after another. At the same time, resources that are not dependent on each other can be created in parallel.

For Aliyun ROS and AWS CloudFormation, templates can be used to create a resource stack. A resource stack is a set of resources on the cloud, that is, the infrastructure defined in the template. Later, when you need to add / delete / change some resources, it is also achieved by using templates to update the resource stack.

For Terraform, a configuration file can be used to generate an executable plan that executes the creation of defined resources / components, incremental changes, and ongoing management.

In the case of Pulumi, the code is executed directly for deployment.

This deployment method not only enables resources to be created in a reasonable order, but also improves deployment efficiency, and retries to a certain extent in case of anomalies, which really makes the whole automated deployment process stable and efficient.

Further efficient operation and maintenance based on infrastructure coding

When the entire infrastructure is templated, DevOps becomes easier. We can use version management tools (such as Git) to manage templates that describe the current infrastructure, and use Aliyun Cloud effect / AWS CodePipline/Jenkins to create a pipeline from code submission triggers to manual card points to resource stack deployment, so that the management of the entire infrastructure will become more agile and automated.

Figure 1: flowchart of infrastructure changes

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

After each change of the template, push the branch content of the local warehouse to the remote warehouse and initiate a review

If the review fails, modify the template and re-initiate the review; if the review is passed, the pipeline will be triggered automatically

The assembly line triggers the manual card point and notifies the superior administrator to check the change. If you do not agree, terminate; if you agree, proceed to the next step

If the template is submitted for the first time, create the resource stack (that is, create the infrastructure); otherwise, update the resource stack (that is, update the infrastructure).

Infrastructure changes and previews

IT infrastructure is not immutable. As the business changes, we may face scaling-down scenarios, or we may face changes in the entire architecture. Fortunately, based on the concept of IaC, we only need to describe the latest configuration of the infrastructure without worrying about how to make changes. But even so, we need to know what changes are going to happen before we change. The change set function of Aliyun ROS and AWS CloudFormation, and the execution plan of Terraform can let us know the content of the change in advance.

For example, due to business changes, a new ECS instance is added on the Aliyun platform based on the architecture in figure 1, and the SLB instance is used to load balance the two ECS instances. After writing a new template, you can use the change set feature to perceive changes. The following is an example of a change in Aliyun ROS:

After confirming that it is correct, you can make the changes. The automated orchestration tool then updates the entire infrastructure to determine which resources are added, changed, or deleted based on changes in the template.

Infrastructure deviation detection and correction

Although automated orchestration tools are used to deploy resources, some people may modify the properties of some resources in the infrastructure in non-standardized ways (such as through the console or API), making the actual situation of the resources different from the resources defined in the template. A good automated orchestration tool not only has the ability to detect differences between the actual attributes of the infrastructure and the attributes defined in the template, but also corrects the template or actual resources based on the difference results to make the template consistent with the infrastructure.

Currently, through the deviation detection capabilities of Aliyun ROS and AWS CloudFormation, you can easily find the differences between the actual resources and the resources defined in the template, and make the template content consistent with the actual resources through the deviation correction function.

Thank you for reading! This is the end of the article on "how to automate the configuration and orchestration of infrastructure coding IaC". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it out for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.