Shulou(Shulou.com)06/01 Report--

This article introduces what the self-developed Devops tool is like, the content is very detailed, interested friends can refer to, I hope it can be helpful to you.

In 2018, in order to solve the pain points in daily operation and maintenance and promote the operation and maintenance work more efficiently, we have developed and improved several tool systems, which without exception have saved us time and improved our efficiency. This article will share and introduce these tool systems.

System introduction

CMDB

CMDB configuration management database is mainly used to record the software and hardware information that we manage and maintain, including physical servers, switches and virtual projects, services, environments and other information that need to be managed and maintained. a popular understanding is that we may have an excel table to record all the projects we maintain, the server resources used by the project, the configuration of the server, and so on. Can be entered into the CMDB system for unified maintenance and management.

CMDB system is the cornerstone of many other systems. It is necessary to provide API to all third-party systems that use basic information to query or modify data, such as providing server information corresponding to the project to continuous deployment tools to push code to the project server, so the data accuracy of CMDB system is very important. At the same time, maintaining basic information in one place can make the whole operation and maintenance system more controllable, more efficient and reduce errors.

Our CMDB system has been online for a long time. Before, it was only used to replace the maintenance information of Excel forms. This year, we added API for him to provide third-party systems with access to basic data. API authentication uses JWT. This article has more introduction about API authentication: Django+JWT implements Token authentication.

Varian

Varian is a modular continuous integration tool developed by us, which is mainly responsible for the process of the project from the source code to the final deployable program. Now that most of the projects have been deployed by Docker, varian will be responsible for the process from the source code to the final packaged project image and uploading to the image repository, which involves compilation, merging, compression and other operations. This article describes in detail the working process of our varian: exploring varian: elegant release and deployment program

The core logic of varian is to split each small step in continuous integration into independent classes or methods, and finally assemble different classes or methods according to different project types, so that different types and different technology stack projects can share the same set of continuous integration programs, reduce code redundancy and improve availability.

Nova

Nova deploys continuously and cooperates with varian to do the whole launch process. Nova is mainly responsible for the process of pushing the final deployable program or Docker image to each node online to update. Because the online environment is relatively complex, such as CVM, Docker container, private cloud, public cloud k8s, etc., compatibility is made at the nova layer.

Nova accepts only three parameters, 1. Project name, 2. Deployment environment, 3. Deploy the version number, call the API provided by CMDB according to the project name and deployment environment to determine which node the project is finally pushed to, and pull the replacement code repository code or image repository image according to the version number.

Expansion, rollback, restart and other operations can be done automatically through the nova system. This article introduces more details of continuous deployment: the continuous deployment optimization practice of Docker environment.

Kerrigan

In the whole process of release and launch, in addition to code changes, changes in configuration files and databases are usually involved. in order to solve the problem of automatic update of configuration files, we have developed a kerrigan system. This article has an introduction to the implementation details of the configuration center: the landing configuration center of small and medium-sized teams.

The underlying implementation of kerrigan is based on etcd+confd, which mainly implements the function of modification on the web side and automatic update on the server. Kerrigan can also manage different types of configurations in multiple environments, especially good at file-based configurations (different from the commonly seen KV-based configuration centers, which are more friendly to operations and maintenance), such as managing configurations such as nginx,tomcat, while recording the modification history of configuration files, quickly rolling back configurations, and supporting configuration file comparison Only modify, save, postpone release and other functions

Because we have many projects, there are a lot of rules in the nginx of each project. Based on Docker, every rewrite update needs to be repackaged and released. It is cumbersome to repackage and publish. This problem is solved effectively after using kerrigan.

Overmind

Database operation and maintenance system overmind, in addition to solving the last link of database changes in the process of release and launch, we also integrate some other practical functions, such as SQL audit, SQL query, automatic data guide work order system, password table and so on.

The first version of overmind mainly integrates inception for SQL audit and execution, which helps us to automatically deal with the changes of online database. This article introduces: small and medium-sized teams quickly build SQL automatic audit system.

After the completion of the first version, we internally promoted the use of development and testing, collected feedback, and added functions such as SQL query and Explain execution plan display on the basis of the first version. Later, it was found that DBA often received the requirement of conducting data between various impassable environments, and developed a work order function to achieve automatic data migration. This article introduces migration: data migration automation of operation and maintenance efficiency.

Later, the way of maintaining passwords by Excel is abandoned, and the function of password table is developed. See the introduction of this article: an example of password management table developed by Django [attached source code]

Overmind is gradually improving, and more functions will be added to improve efficiency based on demand and practicality.

Proxy

Proxy is an agent system, similar to Aliyun's SLB,kubernetes ingress, which is mainly used by the development and test environment

We maintain a lot of projects, each project has multiple different environments, each environment has different domain names, corresponding to different back-end services. In order to simulate the environment that has actually requested SLB agents and to centrally manage these project entries, the previous approach is to point all domain names to a nginx server, and the nginx server proxies to the back-end services through domain name-based vhost Each addition or modification is accomplished by manually changing the nginx configuration file, and now the proxy system has been developed, which can be done quickly and easily through the page.

Wiki

The wiki system was launched 18 years ago, and the goal of standardized, documented, automated and intelligent operation and maintenance was put forward at that time. Documentation is a very important part of the whole operation and maintenance process. Needless to say, its benefits are also very important for us to continuously promote the output of documents.

Of course, in addition to the above systems, some small tools have been developed to standardize management and improve efficiency, which are not introduced here. In addition, we also use a large number of open source software systems, such as Jenkins, ELK suite, Kubernetes and so on.

We know that devops is an idea to automate the whole process from research and development to launch, and it is not a collection of tools or tools. I have been thinking about how to implement devops. In the past 18 years, based on the current environment, we have developed all kinds of tools to help us work efficiently, but these tool systems are relatively scattered and can not form a systematic process. In 19 years, we will practice some ways and methods to connect these tool systems in series to achieve a higher degree of automation. at the same time, we will continue to promote the landing of Kubernetes in a larger range, laying the foundation for the real realization of Devops idea, the whole process automation from development to launch.

About how the self-developed Devops tools are shared here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.