Shulou(Shulou.com)06/02 Report--

At the just-concluded 2020 Global New Generation Software Engineering online Summit, Ren Longtao, a product architect at JD.com Zhaopinyun, who has nearly seven years of experience in the research and development of automated operation and maintenance platforms, shared the topic of "efficient and safe operation and maintenance design and practice landing". This article will review this wonderful sharing and deeply analyze the problems in the process of efficient and safe operation and maintenance of JD.com Zhaopin Cloud, as well as the exploration and landing practice to solve these problems.

A programmer, by his own power, deleted his own company's database, the market value shrank by nearly 2.4 billion yuan, and the direct economic loss reached 150 million yuan.

It's not a joke, it's a true story that happens around us. After the staff of the company's R & D center logged on to the springboard of the company's intranet through their personal API, they carried out malicious damage to the production environment, resulting in the downtime of Mini Program users and the suspension of online business of more than 3 million merchants. It took a week to finally recover the data. This case reflects that there are huge loopholes in the control of operation and maintenance authority in some enterprises, such a dangerous deletion operation that the whole operation can be completed by one person without second confirmation.

This is only one of the security dilemmas that enterprises face in the process of operation and maintenance. at present, enterprises usually face various challenges in operation and maintenance security, such as unverifiable user identity, system account sharing, audit difficulties and difficult operation access control.

With the increasing dependence of daily production and life on the information system, operation and maintenance accidents emerge one after another in recent years. However, due to the sharing of accounts in the process of operation and maintenance, the scope is difficult to control, the password is difficult to manage uniformly, and the division of personnel authority is not clear, so it is very difficult to audit and trace after the accident.

On the other hand, China has gradually promulgated a series of laws and regulations for data security, which put forward strict security compliance requirements for enterprises. For example, the Network Security Law requires that logs be kept for not less than 6 months, and network security measures must be taken; "Equal Protection" 2.0 requires enterprises to authenticate the identity of users, such as access control of users, the principle of minimum authorization, operation and maintenance operation complete audit, regular data backup, etc.; operators need to meet the regulations of the telecommunications industry; securities and financial industries need to meet the relevant requirements of banking supervision and securities supervision Listed companies should meet the requirements of internal control.

In the face of many challenges in the process of operation and maintenance, JD.com Zhilian Cloud, after years of continuous exploration and practice, gave its own answer-the operation and maintenance fortress machine. The operation and maintenance fortress machine mainly includes two functions: operation and maintenance management and audit, which can carry out unified identity authentication, unified asset management, unified access authorization and whole-process operation and maintenance audit.

The operation and maintenance fortress machine is suitable for a wide range of scenarios, and it can be used in almost all enterprises or institutions that need safe operation and maintenance, such as the Internet, finance, government, enterprises and institutions, and so on. Especially after the outbreak of COVID-19 at the beginning of this year, many enterprises require their employees to work remotely from home. In this case, for enterprise operation and maintenance personnel, VPN+ operation and maintenance fortress machine is undoubtedly the best choice.

When it comes to the operation and maintenance fortress machine, we need to review several important stages of its development. At first, the operation and maintenance fortress machine appeared in the form of hardware, and the advantage was that it had its own system. But the disadvantage is also obvious, the product is relatively solid, the upgrade is difficult, and the expansibility is poor.

Until 2000, the software fortress machine appeared, which solved the defects of the hardware fortress machine, but the advent of the cloud era posed a series of challenges to the software fortress machine:

The first challenge is a multi-cloud architecture. In the cloud era, the target assets have changed greatly, and the enterprise IT assets are rapidly clouded. Especially in the current non-regulatory industries, non-financial industries in the cloud is making rapid progress. With the rapid expansion of business, users will always face some security compliance requirements. At this time, users will find their assets more and more scattered and more and more difficult to manage. The continuous promotion of hybrid cloud also makes the management of IT infrastructure more and more complex. Enterprise infrastructure includes traditional KVM, private cloud, public cloud and other different types. The fortress machine needs to adapt and manage different IT components, and conduct a unified security audit.

The second challenge is a change in the way assets are managed. Traditional fortress machines are usually entered by hand, or imported through text import or scan IP. Because of the high network complexity in the cloud era, the network deployment mode of the traditional fortress machine is faced with great challenges. The assets of the cloud era are usually distributed under different clouds, different VPC, and different subnets. How can the fortress machine better adapt to this network environment?

The third challenge is the need for a mature highly available disaster recovery deployment architecture. Highly available distributed technology provides key technical support to support the normal operation of the platform. Disaster recovery system still needs to ensure the security of user data and provide uninterrupted application services in the event of power outages, communication failures and hardware and software errors. The fortress machine needs to provide continuous availability and quickly switch disaster recovery to ensure uninterrupted operation and maintenance capabilities within the enterprise and effectively support the normal development of the enterprise business.

The fourth challenge is the need to support horizontal expansion. At present, enterprise virtual assets are dynamic, and with the growth of subsequent business, the number of assets will continue to grow. The large and fast-growing IT assets require Fortress to be more scalable in terms of asset management to cope with sudden business needs.

In the face of the complex operation and maintenance scenes and requirements of the cloud era, JD.com Zhaolianyun formulated a principle when designing products, that is, "the fortress machine we need is the fortress machine we want to make." The product demand of operation and maintenance fortress machine comes from thousands of software engineers throughout JD.com Zhaopinyun, who have thought deeply about cloud computing and judged the future development trend in daily software development, launch, and operation and maintenance.

Based on these considerations, the design of JD.com Zhaopin Cloud Fortress needs to include the following four functions:

First, support mainstream cloud vendors and private cloud frameworks.

Second, a reliable security mechanism. Security is the lifeline of an enterprise, which must be considered and guaranteed as the primary core element.

Third, the ultimate user experience. JD.com is an Internet company, which requires products to be positioned by the requirements and experience of Internet products at all times. The best experience of the fortress machine should be that although users are using the fortress machine products in the process of development and operation and maintenance, they do not feel the existence of the fortress machine.

Fourth, one-click deployment is supported. JD.com Zhaopinyun Viborg machine can support minute-level deployment, can complete the import of asset information, user information and account information with one click, and is simple and fast from deployment to normal use.

Combined with these requirements and product design principles, JD.com Zhilian Cloud found that low-coupling, hierarchical, distributed architecture has become the best choice for operation and maintenance fortress machine.

First of all, let's take a look at the hierarchical architecture. JD.com Zhaopin cloud storage layer uses cloud disk, etcd, es and oss; data control layer. The api server; core layer of K8S is the API service Bastion developed by JD.com Zhaopin Cloud, which is used for a series of core API management such as core authentication, asset management and user management. The above is the access layer, including a self-developed user-friendly SSH interface Relay, the front end provides a user management interface, and Web Teminar, which can be operated through the browser. These three modules provide users with a rich entrance to the operation and maintenance.

Each layer of the architecture is decoupled and can be deployed in theory, and each layer can be expanded horizontally according to business needs. Currently, JD.com Zhaopin Cloud centralizes all modules in one image for containerization deployment, and the number of deployments can be expanded according to needs. For example, you can have four containers and the same image to form a highly available version of the instance. This high-availability version of the instance is distributed and deployed in different regions, and the lower-level etcd will also form a cluster to ensure the high availability of the overall architecture. The whole architecture is considered from the cloud native point of view, which is more in line with the requirements of the cloud era for fortress machines.

At the same time, JD.com Zhaopin Yunyun Viburg bastion machine is also fully in line with the design of the fortress machine for safety 4A principle:

First, unified identity authentication. Users' roles and permissions should be uniformly managed to achieve separation of powers and perform their respective duties; support SMS, Google authentication, LDAP, AD domain and other authentication methods to facilitate interfacing with users' existing operation and maintenance methods; support two-factor verification; support batch user management, such as importing sub-users through IAM interface, importing users in batches through files, or classifying users through user groups.

Second, unified asset management. It includes account number, modular management, support password, public and private key account, SSO single sign-on without user perception. Assets also support batch import and grouping management, and can be authorized according to the host group. The account number of the asset supports automatic encryption, and the user designs the encryption plan and executes it on a regular basis. After the password is changed, the user has no perception and can log in automatically, which solves the difficult problem of account management and reduces the workload of operation and maintenance.

Third, unified access authorization. Explain in one sentence who accessed which machines at what time and from where, and what he did, what he didn't do, and what he did on the machine. You can know exactly who this person is through access authorization.

Fourth, rights management. The access strategy of fortress machine mainly includes IP restriction, access time limit, high command restriction, secondary authorization and so on. Through the restrictions of these rules, we can ensure that in the whole process of operation and maintenance, users must operate under the assigned authority, and there will be no ultra vires operation, so as to ensure the security of the overall operation and maintenance work.

Finally, the whole process audit of operation and maintenance. Operation and maintenance audit is the last line of defense to ensure the safety of operation and maintenance. through the whole process of operation and maintenance audit, it can quickly locate the problem and determine the responsible person. This helps to quickly resolve the problem and restore normal service. The whole-process operation and maintenance audit is mainly realized through three schemes: full-process video recording, instruction recording and command retrieval. The files of the full-process video recording should be downloadable, backup and playable; the instruction full-process recording is mainly aimed at character-based operations; command retrieval needs to support command-level full-text retrieval, so that when there is a problem, you can query and retrieve the relevant command set and quickly locate the problem.

Combined with JD.com Zhaopin Yunbao bastion machine, JD.com Zhaopinyun is using a set of machine authentication management scheme based on service tree authorization, which solves the complexity of traditional fortress machine role rights management and the isolation of bastion machine and operation and maintenance system. Operation and maintenance personnel can synchronize user information, decision-making information and asset information in a unified operation entrance, and the whole process is screenshot, traceable, and a professional audit system that meets the 4A standard, which supports JD.com Zhaopin Cloud's internal development and operation and maintenance work safety and control.

Externally, JD.com Zhaopinyun provides users with two forms of public cloud products and privatization programs. JD.com Zhaopin Cloud's public cloud architecture uses the second-generation Intel ®strong ®scalable platform, which can obtain a cloud infrastructure with strong performance and easy to use, reduce the complexity of cloud deployment, and can be used to build a unified cloud data platform to provide comprehensive acceleration for data processing, analysis and AI. It also helps to achieve elastic expansion, stability, reliability, cost reduction and efficiency efficiency of the cloud by combining automation and intelligent management features. Directly using the operation and maintenance fortress machine on JD.com Zhaopin Cloud can help enterprises to start efficient operation and maintenance on the cloud immediately.

Through JD.com Zhaopinyun privatization operation and maintenance fortress machine plan, a bank in Hong Kong has built a complete and advanced operation and maintenance audit management system. With the distributed deployment scheme of the fortress machine, unified management, unified authorization and unified access are realized in the distributed assets of the local IDC and multiple public and private clouds; at the same time, combined with its own zero plug-in access capability in the fortress machine distributed deployment scheme, users can simply and efficiently access the host assets anywhere using only mainstream browsers.

JD.com Zhaopin's good operation and maintenance service gives the bank's operation and maintenance audit management system a sustainable platform evolution ability, and the bank's customers can get the latest software version and software patch upgrade service in time. And the first time access to the original factory fault troubleshooting, emergency assistance and other professional services, the stability and security of the system are effectively guaranteed.

In the financial field, banks not only need to consider the safety of operation and maintenance, but also need to consider many factors, such as risk control, credit, marketing, technology foundation and so on. As the most industrial technology service provider, JD.com Zhilian Cloud is committed to creating a secure, credible and intelligent supply chain financial system ecology.

In the field of risk control, JD.com Zhaolianyun is based on JD.com and external big data, based on big data, artificial intelligence technology and algorithm for licensed Xiaojin, banking institutions, micro-installment credit and other enterprise institutions to provide a complete risk control solution system. And for small and medium-sized bank customers with the theme of city commercial banks and agricultural businesses, consumer finance, trust and other financial institutions, provide one-stop full-process online credit comprehensive services, rapidly improve the quality of online credit assets, and establish active risk control ability. the system is suitable for risk control and management in the whole life cycle of retail credit business scenarios.

In addition, JD.com Zhaopinyun also stems from the ability precipitation and best practices of JD.com system for many years, providing an one-stop R & D operation and efficiency improvement platform for the financial industry, helping customers build an integrated framework of R & D and operation, covering the whole process of R & D, testing, operation and maintenance, achieving efficient integration of R & D resources, improving development efficiency and delivery quality, and helping financial enterprises to improve efficiency and business innovation.

Click "read the original text" to learn about Jingdong Yunbao base machine and related operation and maintenance products immediately!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.