Shulou(Shulou.com)06/01 Report--

This article describes in detail the corresponding analysis and solutions to how to use Docker+Rancher to build an automatic release pipeline for Unicom data, hoping to help more partners who want to solve this problem to find a simpler and easier way.

Hello, everyone, today I will share the experience of using Docker and Rancher to build an automatic release pipeline for Unicom data. I will introduce the process and scheme design of automatic data release, the experience we have learned from some of the pitfalls we have stepped on, as well as the operation status of our automated distribution pipeline.

I. the demand for communication data and the reasons for choosing Rancher

Tonglian data is a newly emerging financial technology company in recent years, the main goal is to combine big data, cloud computing, artificial intelligence and other technologies and professional investment concepts to create a world-class and revolutionary financial service platform. This involves several key words: big data, cloud computing, artificial intelligence, these hot words have been mentioned by many enterprises, so I will not repeat them.

As a start-up company, Tonglian data will also encounter many problems that start-ups will encounter, such as too many products, and every year many products will be overturned and restarted, so they will encounter all kinds of problems. For example, when an application is developed, it involves issues such as how much CPU is needed and what language to program. Developers will not communicate with the background personnel in advance to produce a product and directly put the operation and maintenance personnel online.

Based on this background, we urgently need to get through the online pipeline, because in the foreseeable future, a large number of new applications will appear every year, and how to get through the online pipeline needs to be solved urgently.

The first thing we do is to do continuous integration. As our company produces hundreds of new projects every year, the language, framework and deployment methods of each project are different, the release process is long and inefficient, and the deployment of applications takes up a lot of time for operation and maintenance personnel.

We decided to solve this problem with containers. After evaluating several suppliers, we chose Rancher. The main reasons are:

First of all, the user interface of Rancher is very simple. I believe people who have used Rancher will have this feeling. It does not require much professional knowledge and is easy to use. Secondly, Rancher is also very simple in deployment and can be deployed with one click. Rancher also provides good API support to facilitate integration.

Second, the process of automated release

Then we started to build our own CI/CD. At that time, we encountered a lot of difficulties in the process, the following figure is already simplified, in fact, there are a variety of bifurcations and branches:

As far as our original process is concerned, the first part of the process is research and development; then it is deployed in the QA environment, which requires people to deploy, usually operation and maintenance personnel, but operators are generally reluctant to do this. After the deployment is completed, enter the test phase of the QA environment, and notify the developers and testers to test. There may be delays in the process, because the testers may be busy with other things and cannot be tested immediately; after the QA environment tests are passed, they will be deployed in the STG environment; and then the STG environment tests will be carried out, which may be cycled many times. Then enter the security testing phase; after the test passes, there is a formal package preparation process before it can be deployed in production. Even this simplified process is very complex, and it involves a lot of offline communication, so it is impossible to be efficient.

After using Rancher, the original process is greatly simplified, as shown in the following figure:

The first step in the process is continuous integration, which means that after the developer has written the code, the automatic compilation can be triggered directly through CI,CI, and then the script can be automatically deployed, and the test environment is ready. To put it simply, every time the developer submits the code, the test environment is always in a ready state, at this time you can directly enter the testing phase, the whole process is offline, there is no need to go through any process, and it is all automated.

After the tester completes the test, the tester carries on the STG environment test, because the background has already completed the docking with the Rancher and realized the automation, which gives the QA environment test the powerful automation ability that has never been seen before, which means that the QA environment test can be automatically docked to the STG environment test. After the test is passed, we enter the security testing phase, which is required by the company and can not be avoided. After passing the security test, we will enter the production deployment. Previously can not bypass the offline communication steps and some deployment can be omitted, the whole process is optimized and concise, and the efficiency is also improved.

CI/CD may be very simple to say, such as PUSH code, QA environment is automatically ready. However, this is not the case in practice, and there are still many problems to be solved. For example, the branch model of development involves which branches of PUSH can be deployed when developing code, or all branches of PUSH can be deployed?

Third, develop the branch model

At that time, we thought that the best thing was that any branch of PUSH could be deployed, which would be very convenient. But then it was found that this method did not work, would cause confusion, and was difficult to manage. A The Successful Branch Modeling branch model of our Git is similar to this, which specifies a develop branch, a branch of type feature, a branch of type release, a branch of type hotfixes, and a branch of type master.

In normal development, developers often cut out a feature branch on the develop branch, for example, to develop a story that contains a lot of functions, then everyone cuts a story, where the story has its own ID, and then go to the development, after the story development, and then merge it over. In the end, we only choose one line to do CI, contemporary code PUSH or merge into the develop branch, we do this for you, and in this feature branch, we deal with it separately.

Feature branch means that when users submit a feature branch, we will deploy another set for you, one for each feature branch, which means that each story can be tested separately, and finally merge it into the develop branch. Then when testing, the test can decide which branch line it is going to test on according to its own requirements.

For example, if the user only cares about story An in the A test, test it in story A, a branch test environment. After all the story is merged, a centralized test is conducted. After the test is passed, the branch is cut to a branch of release when the test is passed, and then the official package is released on release, allowing QA to continue testing in the STG environment, as seen earlier in the flowchart. The branch model is very messy, and in order to do CI, we define each branch with the developer, and each different behavior corresponding to each branch is also defined, which is very useful in the chaotic branch model.

IV. Version number rules

In order to do CI, the rules for version numbers must be consistent, and if each team's version number is named differently, the matching rules will be very troublesome and confusing. Later, we chose the rule of a version number of Semantic Versioning, which is what time X, which is a common version number naming method, this version number contains a standard document, the document describes the specific definition of this version number, the first is called MAJOR, the second is called MINOR, the third is called PATCH, and you can add a variety of your own version numbers.

5. CI trigger path

Next I will introduce the trigger path of our CI-Git push,push to develop branch, Git push will push to the server of Gitlab, and then call Jenkins,Jenkins through webhook to bulid the package. We originally wanted to call Rancher's API through Jenkins, but later we found that there was a gap in directly calling Rancher, and the process was not so smooth. In order to solve the problem of gap between Jenkins and Rancher, we installed a Ponyes software between them.

Why do you need Ponyes as a middle tier? What value does it provide?

(1) dynamically modify the version number

First of all, it can solve the problem of dynamically changing the version number. When we use Rancher, we find that Rancher's store is very easy to use. We can define some things in the store, and then QA can deploy an application as long as we fill in a few parameters. Before there is no Ponyes as an intermediate layer, we must find operation and maintenance personnel to do it, and the process is more complicated.

To use the Rancher App Store, we also define a template for the App Store so that we can generate a real deployable application every time we PUSH code. However, there are still some problems with the version number. Every time we PUSH code, when we go to Jenkins, we will upgrade the version number of Jenkins according to the number of reads, such as 1.0.1.0-1, the first time is-1, and the second time, that is,-2, the version number of Jenkins changes accordingly according to the number of times of reading. At this time, the app store will change with it.

So, we made a template in such a way that in Ponyes, each Jenkins can get the corresponding version number, and then inject this version number into it through variables:

Sample: image: {{REGISTRY}} / automation/auto-sample: {{m ['auto-sample']}}

This process also involves registry, because the QA environment and the STG environment are completely separate, and when rendering the template, we need to know whether to send it to the QA environment or the STG environment, so as to change the address of the registry accordingly. In this case, the problem of changing the version number mentioned above will be solved.

(2) Multi-service management

There is also a thorny problem, that is, what if there are multiple services in a stack? For example, a small team may have a total of several people, each responsible for several projects, and the relationship with microservices is somewhat similar, then a stack may have several services, most typically front-end, back-end, or other middleware, each of which is a service.

At the beginning of deployment, these services in Rancher are also managed by a stack, and multiple services are faced with the problem of how to manage them. For example, a stack has three ABC services, and when service C updates, the entire stack should also update the version number, because in Rancher, after stack is deployed, there is a yellow button for update available. If there is a new version, click this button, you can upgrade the stack, and you must upgrade the entire stack instead of just one service. At this time, you need to manage multiple services. Ponyes records the relationship between the service and the stack. Any service update will also trigger the update of the stack. The update method is that each service is updated with the version number of the stack + 1.

(3) release multiple versions in parallel

Then there is a more serious problem: what should we do if multiple versions are released in parallel? For example, we ABC three modules, A released 1.0 and 2.0 Magi B released 3.0, 4.0 Magi C released 5.0 and 6.0. If we release 5.0.2 of a C application, the question of what the corresponding AB module number should be chosen has been bothering us for a long time.

There are several ways to solve this, such as using something to record the relationship between c5.0 and a version number of AB, which means that users can customize it. However, there is also a problem, users need to manage the relationship between their version numbers, for a long time, they may confuse or make mistakes in the relationship between version numbers, or the relationship between version numbers, and then go online to the production environment, with more serious consequences.

Another way is to statically record what version of AB was when the user's C version number was last released to 5.0. But in this way, the system will become quite complex and error-prone.

Finally, we choose to remove the ability to release multiple versions in parallel and only support the release of a single version, which means that if it is to be released, it must be the latest version, and the historical version can be processed manually in another way, so that the system will be simpler, and it is not error-prone, and there is no need for users to maintain the relationship between a version number. All these functions are solved with the help of Ponyes written by ourselves.

Rancher 2.0 also mentioned such a function of CI/CD, in the actual process, will encounter a very real problem, from code research and development to the deployment of the entire production environment, there are many links, the specific situation is also very complex, and the role of CI may only end in QA, behind the link there will be new problems, then we need a system to run through the entire life cycle management, the role of Rancher lies in this.

VI. Three experiences summed up by stepping on the pit of the body

(1) how many Rancher environments are deployed?

This issue seems to be very small, and it has aroused a lot of discussion among us at first. Initially we deployed only one set of Rancher, in which the environment was used to distinguish between QA and production in Rancher. The easiest way to deploy a set of Rancher is, but there is a serious problem: what if Rancher is upgraded? This set of Rancher manages both QA and production, and the production environment needs to be upgraded when upgrading. If bug appears at the moment, the problem will be very serious.

Later, we broke it into four sets, and the Rancher platform itself had to have an upgrade environment. It is recommended that you deploy multiple environments (at least two) when you deploy Rancher. We actually have one for each environment of dev, QA, staging and production, with a total of four.

(2) configuration item explosion

The second question is about configuration items. We started with Rancher compose, which is very simple and powerful. We click "deploy" in Rancher Catalog, and all the configuration options pop up. We just need to click on something to deploy a very complex application. Later, we found that there are more and more configuration items (even as many as hundreds). This makes it difficult to display them on one page, and hundreds of configurations make it impossible for us to fill in, and operators can not deploy them successfully, so we are faced with the problem of configuration item explosion.

Our solution experience is that in the container platform, configuration items must be managed centrally. We use consul to manage all the configuration items, and every time the container starts, we go to consul and pull the configuration to the corresponding container. In this way, the container can drift on any platform. In addition, the configuration item itself has a copy in the original server. If we modify the original configuration item in copy, we can deploy another instance. Therefore, all configuration items must be managed centrally.

(3) Pan-domain name + Rancher LB

Each of our businesses has a web service and we have to apply for our own domain name. We have hundreds of projects online every year, which means there are hundreds of domain names to apply for. In addition, these domain names need to be used in development, testing and production environments, so we have to apply for nearly 500 domain names every year. This is a terrible thing.

Later, we used the pan-domain name method. For example, use the domain name of * .sub.example.com to CNM directly to one of the hosts in the Rancher environment. Then set its LB,LB on Rancher and it can be distributed across all hosts, each with the same LB. This domain name points to any host and it can work.

For example, in the QA environment, the following domain names are serve on the LB. If the host goes down, the container itself will start, and the entry problem can be easily solved by changing the * quantity. If it is a production environment, you can add a layer of nginx on it, with three upstream, kill any one, and enter through the other two entrances.

In the method of using pan-domain name, after configuring the pan-domain name, you don't have to apply for a new domain name by adding any domain name to Rancher LB. Instead, you can write 123.sub.example.com directly, and then configure it directly on LB. After the configuration, the domain name can be used, and there is no need to go through the application process.

This is the answer to the question about how to use Docker+Rancher to build an automatic release channel for Unicom data. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel for more related knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.