Shulou(Shulou.com)06/02 Report--

In this issue, the editor will bring you about how to treat DevOps. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

What on earth is DevOps?

DevOps (a combination of Development and Operations) is a culture, sport, or practice that emphasizes communication and cooperation between "software developers (Dev)" and "IT operations and maintenance technicians (Ops)". Build, test, and release software more quickly, frequently, and reliably by automating the process of "software delivery" and "architecture change". Wikipedia

DevOps is a cultural shift, or a movement that encourages better communication and collaboration (that is, teamwork) to build more reliable and better quality software more quickly. -- Mike Kavis

Mike Kavis, vice president and chief architect of Cloud Technology Partners in the United States, also described in more detail that DevOps is the development of the software development life cycle (SDLC) from waterfall to agile to lean. DevOps goes beyond Agile and its focus is on removing waste from SDLC. In general, the forms of finding waste or bottlenecks include: inconsistent environments, manual build and deployment processes, poor quality and testing practices, lack of communication and understanding between IT departments, frequent interruptions and failures, and a full set of problems that require valuable resources and important time and money to keep the system running.

He also sees another repetitive waste: the first step for a DevOps team is usually to decide whether they should use Chef or Puppet (or anything else that is hot, such as Salt, Ansible, etc.). They haven't even defined the problems they intend to solve, even if the tools they have at hand can solve them. These teams often nervously build thousands of lines of scripts, but this raises the question: "is it our job to write Chef scripts or to get to market faster with better, more stable products?" . In most cases, these teams push themselves into a corner, a large number of proprietary scripts actually increase the waste of the system, and the driving force behind the DevOps movement is to remove waste from the system, which these teams do not do. Mike Kavis original text

At present, there are too many views and definitions of DevOps, but they all share a common idea: "solve the once insurmountable gap between developers and operators, and enhance the communication and communication between developers and operators." Personally, I think DevOps can be expressed in a formula:

Change of cultural concept + automation tools = constantly adapt to the rapidly changing market

Its core value lies in the following two points:

Deliver faster and respond to changes in the market.

Pay more attention to the improvement and promotion of the business.

When we understand what DevOps is, why do we need it? What benefits does it bring to us?

Why do you need DevOps

The speed of change in today's world is different from that in the past, and every time it goes through a subversive technological revolution, it has brought profound changes to the world. Emerging technologies such as big data, cloud computing, artificial intelligence, VR/AR and blockchain promote continuous changes in the world. How to deal with such a VUCA era, so that we can respond quickly when the environment changes?

V=Volatillity (variability) is the essence and driving force of change, and it is also driven and catalyzed by change.

U=Uncertainty (uncertainty) lacks predictability, expectations of surprises and understanding and awareness of things.

C=Complexity (complexity) enterprises are plagued by all kinds of forces, factors and things.

A=Ambiguity (fuzziness) vagueness of reality is the source of misunderstanding and the mixture of various conditions and causality.

Next, I will analyze and introduce how to change from the two aspects of "product iteration" and "technological innovation".

Product iteration

Whether we are doing the Internet or games, in fact, we are ultimately making a product, a product that users like. Jobs famously said, "consumers don't know what they need, and it's not until we come up with our own product that they realize that this is what I want." So Master Joe was able to design the final effect of the product from the very beginning, and then iterate step by step according to the parts, the steps can be shown in the following figure:

There is only one Jobs in the world, but in our real product iteration, the conversation is as follows:

User: I usually walk to and from work. I have to walk five kilometers every day. It's so hard. Is there any way to help me design a tool to solve my pain point?

We thought about it and thought it was not very difficult, so we could try it, so we discussed-> Design-> Development-> Test-> delivered a skateboard to the user.

User: this skateboard is not easy to operate. Can you give me a handrail?

Then we produced a scooter according to the new needs of the users.

User: the scooter has to slide. Can you let me ride it?

We continued to improve our products and produced a bicycle.

User: the bike still has to climb, and the long journey is also very tiring.

We went on to optimize it and turned it into a battery car.

User: the battery car has solved the demand, but it is not very safe. Can you optimize the product again?

After all kinds of efforts, we finally produced a beautiful car and delivered it to the users, which finally satisfied the users.

The real user doesn't know what he wants at first, but he doesn't know what he doesn't want until he sees our product.

That is to say, if the real product iteration is so tortuous and iterative, is there any way to deliver value quickly and respond flexibly to changes? The answer is DevOps, which is a business goal-oriented best practice that contributes to business success.

Product iteration requires DevOps, so technological innovation promotes the rapid development and implementation of DevOps. Let's take a look at how technology supports product iteration and continuous innovation.

technological innovation

In the previous system, the business was single, the logic was simple, and the number of users was small, and the size of the project team was generally 10-30 people. The current system has to face customized recommendations from different users, the Internet connects people and people, people and things, as well as things and things, the business is becoming more and more complex, more and more functions, if the whole system is coupled together, it is bound to affect the whole body, resulting in the maintenance of the system is very difficult.

Therefore, the IT technology architecture is constantly changing and innovating with the complexity of the system, from the early All In One of all services to the current micro-service architecture, from pure manual operation to fully automated processes, from a single physical machine to the cloud platform. The following figure shows the change of IT technology innovation:

Now that DevOps has become the trend of development, how does it land?

How to realize the landing of DevOps

The truth of knowledge is practice, and the insight of action is knowledge-- the Biography of Wang Shouren in the Ming Dynasty

Here I quote a famous saying by the sage Wang Yangming, who advocates "the unity of knowledge and practice". In popular terms, it is necessary to combine theory with practice in doing things. When we achieve the landing of DevOps, we must follow the way of "the combination of theory and practice". Theory is the guiding ideology for us to do things, and practice is the specific way to do things. Then I will learn from how I in the company is in accordance with the combination of theory and practice to promote the implementation of DevOps.

Implement the guiding ideology of DevOps

First of all, we still have to go back to what DevOps is, and if you forget, you can go back and review it again, including the DevOps formula that I summarized.

In fact, the core idea of DevOps is: "deliver value quickly and respond flexibly to change". The basic principles are as follows:

Efficient collaboration and communication

Automate processes and tools

Rapid and agile development

Continuous delivery and deployment

Keep learning and innovating.

However, how are these basic principles closely related to project research and development, that is, how are they reflected in all aspects of our development process? Take a look at the following introduction from "success with enterprise dev-ops-whitepaper":

Agile management: a trained agile development team is the key to the successful implementation of DevOps.

According to Conway's law, the product developed by the software team is a reflection of the company's organizational structure.

Therefore, adjusting the organizational structure according to the situation of the company is the first condition, which will directly affect the requirements, the efficiency of the design and development phase, and the cost of communication.

There is a good formula for calculating the communication cost of the team in the "Man-month Myth": communication cost = n (nmur1) / 2, where n is the number of people, so the communication cost will increase exponentially with the increase of organizational personnel. How to divide small and fast agile teams, which I will explain in detail in the section "how to implement DevOps" later.

Continuous delivery deployment: automates the construction, deployment, testing, and release of applications.

Through technical tools, the traditional manual operation can be transformed into an automated process, which will not only improve the efficiency of product development, operation and deployment, but also reduce errors and accidents caused by human factors, find problems in advance and solve them in a timely manner. This also ensures the quality of the products. The following figure shows the process of DevOps automation:

This diagram is from my new book, distributed Service Architecture: principles, Design, and practice, which also details continuous delivery deployment.

IT service management: sustainable, highly available IT service is a key element to ensure the normal operation of the business, which is integrated with the business.

IT service management (ITSM) directly affects the whole life cycle of product operation. Traditional IT service management (like ITIL) does very well in production, but it is too tedious for DevOps, so it is necessary to create an ITMS for DevOps that only pays attention to business continuity. It only needs few necessary resources to provide services for the corresponding business. ITMS considers it more from a business point of view.

Note: explain what is IT service management (ITSM) in vernacular. It is a mode that traditional "IT management" shifts to "IT service". The former may pay more attention to specific server management, network management and system software installation and deployment. The latter pays more attention to the standardization and standardization of the process, clearly defining the goal and scope of each process, cost and benefit, operation steps, key success factors and performance indicators, the responsibilities and rights of the relevant personnel, as well as the relationship between each process, such as the establishment of online accident resolution process, service configuration management process, etc.

It is not enough to have processes alone, because processes are mainly used internally by IT service providers, and customers are not interested in them, so you need to package these processes into specific IT services on demand and provide them for customers to use, such as purchasing a virtual CVM on a cloud platform.

Lean management: establish a pipelined IT service chain, bridge the gap between development and operation and maintenance, and realize the agile mode of integration of development, operation and maintenance.

Lean production mainly comes from the production philosophy of Toyota mode of production (TPS), which is famous for reducing waste and improving overall customer value. It mainly uses the optimization of automated processes to improve productivity and reduce waste. So the essence of lean production is just-in-time (JIT) and automation (Jidoka).

JIT (Just In time): JIT is described in one sentence as consuming the least necessary resources to produce and deliver the right parts in the right quantity. Working in this mode can minimize inventory and prevent premature or overproduction. Most companies prefer to use inventory to avoid potential downtime risks, while Toyota does the opposite. Make a timely and effective response to the problems arising in production by reducing inventory. Of course, the JIT model is a considerable test of problem-solving ability, in the case of insufficient capacity, there will be a considerable risk of disconnection.

Jidoka (Build in quality): automation, which means "customization" in Japanese, literally means "automation" in Japanese, while in Toyota TPS system, the word "action" is specially added to the word "human" and becomes "automation". In other words, the process control that TPS/ lean production is eager to produce can be as intelligent as "human" and automatically shut down in abnormal circumstances at the first time. This automatic stop function can prevent the bad parts from flowing downstream, prevent the machine from causing damage in the wrong production state, and also make people better carry out fault analysis in the current error state. When the equipment can automatically analyze the fault, the "people" of the supervision machine can be truly liberated and the labor cost can be saved.

-- from Zhihu

The following image shows the lean cabin in Toyota's TPS (Toyota Production System) manual:

Lean software development is the application of lean production and practice in the field of software development, which is summarized as follows:

Eliminate waste

Reinforcement learning

Delay the decision as much as possible

Release as soon as possible

Devolve power

Embedded quality

Global optimization

Lean management runs through the whole DevOps phase. It encourages proactive identification of problems and continuous optimization of processes, so as to achieve continuous delivery, rapid feedback, risk reduction and quality assurance. Next, let's look at the specific implementation of DevOps.

The concrete method of implementing DevOps

Build a fast and agile team

According to the Conway Law introduced earlier, we can take a look at the current project team structure in the company, as shown in the following figure:

I believe this is not just the structure of our company, but the common hierarchical structure of most IT Internet companies today. They are generally divided into seven departments: product planning, design art, front-end engineer, back-end engineer, test engineer, operation & DBA and market operation. Various departments naturally form a communication barrier wall, mainly in the form of mail and meetings to communicate with each other, inefficient, difficult to change demand, it is difficult to quickly respond to market changes and continue to deliver high-quality products.

So how to adjust the organizational structure to build a Scrum team? (what is Scrum? please refer to Wikipedia.)

We will divide teams according to business functions, establish communication groups, and set up product owners (a planner), Scrum Master (we generally choose testers and test-driven development models) and developer teams (one front-end engineer, one back-end engineer, one test). The final organizational structure and system architecture are shown below:

An efficient agile team is the guarantee of DevOps landing, so the automation process is an effective mechanism to ensure the rapid delivery and continuous deployment of the product. Next, let's introduce how we achieve the automation process.

The process to automate

Just look at the picture and speak. Here is a complete Pipeline of DevOps:

Submit: the engineer tests the code locally and submits it to a version control system, such as the Git code repository.

Build: continuous integration systems (such as Jenkins CI) automatically pull the latest code from the Git code repository to compile and build when a version control system update is detected.

Unit test: after Jenkins finishes compiling and building, the specified unit test code is executed automatically.

Deploy to the test environment: after completing the unit test, Jenkins can deploy the application to a test environment similar to the production environment for testing.

Pre-production environment testing: in a pre-production test environment, some final automated tests can be performed, such as using the Appium automated testing tool, and some tests similar to the actual situation can be tested manually by developers or customer personnel.

Deploy to production: after all tests, you can use grayscale updates to deploy the latest version to the actual production environment.

What technologies are needed to realize the DevOps automation pipeline, and how do they work together? With these questions in mind, I will introduce them in detail in the Technical Stack section of DevOps. Next, let's take a look at the problems encountered in the implementation of DevOps in game projects.

The problems encountered by DevOps in the game project

Problem 1: it is difficult for game services to be stateless.

The game service architecture is still very different from the Internet architecture. Due to the high real-time requirements of games, it is difficult for many game servers to use distributed centralized caching, so it is very difficult to realize the stateless nature of game services. therefore, mature micro-service solutions in the Internet can not be directly applied to games. I will introduce the comparison between games and the Internet in detail later, as well as how game suits are split and decoupled.

Problem 2: manpower shortage

Personnel shortage is actually a common problem in many companies, but in the game companies I have experienced, a mobile game project is usually staffed by 5-6 front-end players, 3-4 back-end players, 1-2 testers and 1 operator. Therefore, it is very difficult to have special personnel to be responsible for the automation process implementation of DevOps, and we can only take the time to work overtime to promote the implementation.

Problem 3: cross-departmental cooperation, high cost of pre-communication and training

In the process of transformation, due to the previous communication and cooperation between departments across the "wall", personnel knowledge system and cognition are different, so team members do not support or cooperate slowly. We can create a tolerant environment for failure by encouraging collaborative responsibility sharing, establishing automated processes, tearing down departmental walls, creating a DevOps culture that rewards proactive transformational behavior, and changing the way risk management is conducted.

Problem 4: the amount of work invested in the early stage is large but the effect is little.

At the beginning of the project, when there is a shortage of staff and a tight schedule, it is necessary to do infrastructure construction, personnel communication training, etc., with high input costs and little effect, which is easy to make the leadership lose confidence. Therefore, the implementation of DevOps also needs to be carried out in stages, gradually improve the process, to meet the current business needs of each stage as the basic principle, which is the principle of refinement software. My work is generally divided into three periods: product prototype period, product testing period and product operation period. (please combine the "DevOps Pipeline" pipeline diagram in the previous automation process section to see the work in the following three periods.)

Product prototype: this is in the early stages of development, so we generally only need to implement Git code repositories, Jenkins CI integration and static code analysis using FindBugs or SonarQube.

Product testing period: on the basis of the above, continue to implement Jenkins integration Gradle to achieve automatic construction and packaging, unit testing, deployment to the test environment and other processes.

Product operation period: finally improve the assembly line, achieve automatic deployment of pre-production environment and production environment, achieve gray update and so on.

DevOps has advanced ideas and perfect ideas, which I think is the best solution so far, but DevOps finally landed thanks in large part to a complete set of technology and open source tools. Next, let's take a look at the technology stack that DevOps has in mind.

Technology stack

If the content of this section involves too much, I will briefly introduce some common open source DevOps technology tools, you can choose to use according to your needs, of course, you can also use an integrated team environment like VSTS (Visual Studio Team Services).

Some of these contents are described in detail in my new book, such as code warehouse management, virtual machine and containerization, continuous integration & continuous deployment tool Jenkins, configuration management tool SaltStack.

Agile management tools

Trello

Teambition

Worktile

Tower

The above tools are more or less the same, just choose one that suits your team. Our company mainly uses Teambition. The screenshot effect is as follows:

Product & quality management

Confluence

Zen Tao

Jira

Bugzila

Among them, confluence and Zen Tao are mainly comprehensive management tools for product requirements, definition, dependence and promotion, while Jira and Bugzilla are product quality management and monitoring capabilities, including test cases, defect tracking and quality monitoring. At present, we use Jira more.

Code warehouse management

Git

Gitlab

Github

Git is an open source distributed version control system; Gitlab and Github are open source projects for warehouse management systems. They use Git as a code management tool and build web services on this basis. We mainly use Git and Gitlab.

Development process specification

Git Flow

Git Flow is a model for organizing software development activities built on Git, and it is a software development best practice built on Git. Git Flow is a set of behavior specifications and tools to simplify some Git operations when using Git for source code control. The Git Flow model is shown below:

Github Flow

Github Flow is a simpler alternative to Git Flow, with only one feature branch and one master branch, simple and clean. The Github Flow model is shown below:

Gitlab Flow

GitHub Flow thinks you can deploy the code directly online by merging feature branches. The Gitlab Flow model is shown below:

Automated build script

Gradle

Maven

SBT

ANT

I currently mainly use Gradle and Maven, while Gradle is a project automation build tool based on Apache Ant and Apache Maven concepts. It uses a Groovy-based domain-specific language (DSL) to declare project settings, abandoning all kinds of tedious XML-based configurations. Oriented to Java applications. Currently, the supported languages are limited to Java, Groovy, Kotlin and Scala.

Virtual machine and containerization

VMware

VirtualBox

Vagrant

Docker

VMware and VirtualBox are the most commonly used virtual machines, supporting a large number of platforms, while Vagrant is a virtual machine running environment management tool built on virtualization technology. Virtual machine management can be easily realized through Vagrant, including establishing and deleting virtual machine, configuring virtual machine running parameters, managing virtual machine running state, automatically configuring and installing all kinds of software necessary for development environment, packaging and distributing virtual machine running environment, and so on.

Docker is an open source application container engine that allows developers to package their applications and dependency packages into a portable container, publish them to any popular Linux machine, and virtualize them.

Continuous Integration (CI) & continuous deployment (CD)

Jenkins

Hudson

Travis CI

CircleCI

Jenkins is an open source software project, which is a continuous integration tool based on Java, which is used to monitor continuous repetitive work. It aims to provide an open and easy-to-use software platform and make continuous integration of software possible. Its predecessor is Hudson.

Travis CI is an emerging open source continuous integration construction project, which is obviously different from jenkins in that it adopts yaml format, which is concise, fresh and unique.

CircleCI is a continuous integration platform for web application developers, mainly providing testing, continuous integration, code deployment and other services for the development team.

Automated testing

Appium

Appium is a mobile automation framework that can be used to test native applications, mobile web applications, and hybrid applications across platforms. Operating systems available for IOS and Android as well as firefox.

Selenium

Selenium tests are run directly in the browser, just as real users do. Selenium tests can be run in Internet Explorer, Mozilla, and Firefox on Windows, Linux, and Macintosh.

Mock test

Mock testing is a test method created by a virtual object for testing some objects that are not easy to construct or obtain during the testing process. The virtual object is the Mock object, and the Mock object is a substitute for the real object during debugging. The Mock frameworks in Java are commonly used such as EasyMock and Mockito.

Consumer-driven contract testing

Contract testing is a test of the interface of an external service that verifies that the service meets the contract expected by the consumer. When some consumers use the provided behavior of a component through an interface, a contract arises between them. This contract contains expectations, performance, and concurrency of input and output data structures. At present, PACT is a streaming consumer-driven contract testing framework.

Automatic operation and maintenance tool

Ansible

Puppet

Chef

The automation of IT operation and maintenance refers to automating the daily and repetitive work in IT operation and maintenance, and turning the manual execution into automatic operation in the past. Automation is the sublimation of IT operation and maintenance work, IT operation and maintenance automation is not only a maintenance process, but also a management promotion process, is the highest level of IT operation and maintenance, but also the future development trend. The following figure shows the comparison of common automated operation and maintenance tools:

Monitoring and management tools

Zabbix

Zabbix is an enterprise-level open source solution based on WEB interface that provides distributed system monitoring and network monitoring capabilities.

ELK Stack log analysis system

ELK Stack is an open source log processing platform solution, and the commercial company behind it is Elastic. It consists of three parts: log collection and analysis tool Logstash, full-text search engine Elasticsearch based on Lucene, and analysis visualization platform Kibana.

Cloud Monitoring (such as Amazon CloudWatch)

Amazon CloudWatch is a service that monitors AWS cloud resources and applications running on AWS. You can use Amazon CloudWatch to collect and track metrics, collect and monitor log files, set alerts, and automatically respond to changes in AWS resources

Game Architecture comparison between Game Industry and Internet Industry

Comparison of project iteration cycle

Iterative Model of the Internet

The development cycle of game projects

From the above comparison, we can see that each requirements iteration of an Internet project can be more agile and faster, because it can split large requirements into multiple small concrete implementations, which ensures continuous delivery and deployment.

Games are more difficult and time-consuming than Internet iterations, because a game can begin to be delivered to users, and the most basic features and methods of play have to be complete before they can be tested and used.

Comparison of request communication mechanisms

In the Internet, it is generally a request-response mode, and usually every request is blocked synchronously, while in the game, it is mostly a request-push mode, which not only pushes itself, but also pushes it to other users in the game. Every request in the game is asynchronous and non-blocking.

Summary: the biggest difference between the Internet server and the game server actually lies in the "state". The state of the game server changes rapidly in real time, can tolerate loss, and requires a large number of broadcast synchronization; the state of the Internet server is generally persistent, does not tolerate loss, and is only related to specific clients. Therefore, it is much more difficult to achieve DevOps in the game than the Internet, and the mature implementation scheme of the Internet can not be completely copied into the game. Next, I will analyze and introduce what the common game service architecture looks like from the game architecture-the source of DevOps implementation.

Analysis of Common Game Service Architecture-DevOps Root

Rest card game

This kind of game generally uses http communication mode, and its architecture is similar to the common web server architecture. It uses redis centralized cache to save the game state, so that it can be loaded through nginx, and the game server can support unlimited horizontal expansion.

Open a room game

Generally speaking, the server side of this type of game is divided into two parts: one is the hall server and the other is the room server. The hall server is a huge broadcast cluster, responsible for less real-time data transmission and query. Room server is a group of small real-time broadcast service processes that can be rented and returned quickly.

In the lobby server, all online players are distributed in one of multiple processes according to their ID. When querying and broadcasting among players, multiple servers are used to operate in parallel, and finally the results are summarized to provide. This kind of operation delay is relatively high, but it allows huge amounts of user data to be stored on different machines, while the room server is only responsible for providing specific game broadcast functions. Once the players enter in a group, the lobby server copies the data to the room server, and the room server is only responsible for these players. At the end of the game, clean up the player data and prepare new games.

Sub-service game

Although the sub-server game model has been in operation for many years, some game operators still hope to allow as many players as possible to play together, because the more active the online games are, the more interactions they generate, and the more fun the game will be. Therefore, it is required to develop a game server model that meets the online interaction of a large number of users-full server model.

The SOA architecture pattern is a classic distributed software architecture pattern, in which RPC runs are used to invoke functions between services, while directory servers such as ZooKeeper are used for service registration and discovery. In this way, the game service is divided into three layers: the front gateway (gate) server, the middle game server (gameServer), and the last database (DB) server. In this way, the network function is extracted separately, allowing users to connect to a gateway server, and then the gateway server forwards the data to the back-end game server. The data exchange between the game servers is also uniformly connected to the gateway service for exchange. All those who interact with DB are connected to the DB server to process on their behalf.

Summary: now the game server is getting bigger and bigger, and different games actually have many of the same functions, so how to split and decouple the game service in order to realize the service of the game becomes very important. Next, I will further introduce how the game service is split.

The decoupling of Game Service-the idea of divide and conquer

Business level split

From the business level, in fact, all RPG games have five basic systems, such as generals, attributes, backpacks, tasks and technology, and the differences of each game are mostly in different play systems, and there are many similarities between business systems and activity systems. The method and ingredient of board surface

Functional level split

From the functional level, such as login system, customer service system, statistical system and monitoring system, we can also be used as general game services, provided to various game projects, so as to achieve the SAAS platform of the game industry.

Multi-dimensional architecture

A set of game platform faces different departments and personnel, so it also needs to be considered and built from different dimensions, so as to meet the needs and convenience of most people as far as possible.

The above is the editor for you to share how to view DevOps, if you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.