Shulou(Shulou.com)06/03 Report--

Author | Li Linfeng

Li Linfeng, author of "the way to Advanced Netty" and "principles and practices of distributed Service Framework", shared the experiences and lessons of large-scale business teams implementing micro-services.

Introduction-background

The service transformation of some complex business systems (such as CRM) involves the cooperation and coordination of multiple business teams, coupled with the complexity of the business itself, micro-service refactoring of existing systems is a challenging task.

The preparatory work before implementation-- the goal should be clear and deal with "giving up and getting".

Lesson

1. The goal of micro-service is not clear:

Each business module is in different technical stages, such as single application, RPC architecture, SOA service and so on. It is not clear that the micro-service goal of each business: whether to improve development efficiency, shorten the new business launch cycle, or simply pursue the advanced nature of technology.

2. Failure to deal with "giving up and getting":

Micro-service is not a silver bullet, it looks beautiful, but it must be handled well if it is to be implemented in large-scale teams and telecom complex business systems. For example, distribution brings about increased delay and reduced reliability.

Experience summary

Be clear about your goals and dare to give up

Preparatory work before implementation-- Micro-service is not just the architect's vision

Lesson

1. The vision is beautiful, but the landing is deformed:

Top-level architecture design only exists in the architect's mind and architecture design documents, design concepts and key architectural attributes, the underlying developers do not fully understand, there is a large deviation when the architecture is on the ground.

2. Large regiments fight on their own:

After the large-scale business team is divided into many small micro-service teams, they need to cooperate and cooperate with each other, and there is a big deviation in the understanding and compliance of architecture design principles.

Experience summary

Unified understanding, organizational empowerment

Preparatory work before implementation-- Technical selection should be comprehensive

Lesson

1. Pay too much attention to the running service framework while ignoring other quality attributes:

In the selection, we pay too much attention to the performance indicators, functional richness, and the degree of intrusion into the business of the micro-service framework, while neglecting the service operation and governance.

2. Ignore the principle of microservice language neutrality:

There are many business modules and the scene is complex. Most of the business is developed by Java, but the business with stringent performance requirements such as billing and pricing still need to use C (C++), GO and other languages. The service framework does not support heterogeneous languages, so there are problems in cross-language calls.

Experience summary

Comprehensive analysis and language neutrality

Cross-team collaboration-there are many problems of interface dependence and schedule coordination

Lesson

1. The API API cannot be provided in time, which affects the development progress of other teams:

A) lack of initial experience: service providers focus too much on the internal implementation of micro-services, rather than giving priority to API that designs micro-services to provide contracted interfaces to consumers.

B) Post-project rule constraints: limit the number of iterative microservice interface changes. The provider is worried that the interface will be changed and refactored frequently, and has been slow to provide the interface contract.

2. There is no effective change notification mechanism:

The microservice interface contract document is managed offline, and there is no effective notification mechanism after the change.

Experience summary

Promote API First design concept

01 Consumer participation:

Promote the design concept of API First, let microservice providers design API with consumers from the perspective of consumers, and publish and control API online at the same time.

02 all online:

Based on swagger, you can design API and publish API online by YAML or JSON.

Cross-team collaboration-interface compatibility issues

Lesson

1. The micro-service version does not have a strong sense of control and can be changed at will:

Delete and modify interface fields at will, resulting in the failure of other team CI construction and the low pass rate of test cases.

2. It is entirely based on management means:

Through management regulations and other constraints interface changes, but the lack of a unified technical support means. For example, add fields, delete fields, different languages, different teams how to ensure the consistency of implementation.

Experience summary

Technical support and management coordination

01

Formulate and strictly implement the "Micro Service Interface compatibility Specification" to avoid incompatible modifications or unauthorized modifications without notifying the perimeter.

02

Technical guarantee of interface compatibility: for example, IDL of Thrift supports adding, modifying and deleting fields, the position of field definition is independent, and the code stream supports disorder, etc. Grayscale routing is realized by carrying major version numbers such as V1/V2 in URL.

03

Continuous delivery of daily build and contract-driven testing of the assembly line to quickly identify and detect incompatibilities.

Microservice implementation-not all businesses are one size fits all

Lesson

1. Research and development stage:

In order to unify the technical architecture, the micro-service transformation is carried out across the board, regardless of the differences between businesses and the pain points of different businesses.

2. Launch phase:

Full-time online at one time, all micro-service transformation businesses are upgraded at the same time, there is no observation period and buffer period.

Experience summary

Step by step and steadily

01 periphery first, then center:

First find some relatively mature, non-core module business to do micro-service transformation, in this process continue to accumulate experience, to prepare for the follow-up core module micro-service.

02 although the sparrow has all the internal organs:

In the early practice of micro-service, in addition to the development tools and running framework, we need to build all the supporting facilities such as micro-service continuous delivery pipeline, micro-service governance framework, call chain analysis and so on.

03 step by step and quantity:

When going online, we can use grayscale routing and other methods to gradually switch the traffic to the new micro-service system.

A Technical detail that can not be ignored-- Micro Service choreography

Lesson

1. The arrangement framework is confused:

Hard-coding / BPMN process engine, script (Groovy, JS) choreography engine, etc., lack of a unified lightweight micro-service orchestration engine (PVM).

2. Micro service orchestration layer:

At the top level, there is a lack of unified design and strategy, which leads to the realization of various business modules. Some are in the background capability center layer, some are in the middle stage, some are in API Gateway, and some are in the client.

Experience summary

Lightweight micro-service orchestration layer

01

Choose the lightweight micro-service orchestration engine (PVM) as needed. The traditional BPMN model is heavier and more dependent in the micro-service era.

02

The choreography of micro-services is suitable to be independent (middle platform) to ensure the atomicity and stability of micro-services in the background, and at the same time to meet various personalized needs of foreground and mobile end.

Technical details that can not be ignored-- distributed transaction

Lesson

1. Distributed transaction selection:

Business compensation, transaction strong consistency (TCC framework) or message-based final consistency scheme, there is no consistency scheme between platform and business, and between different business teams, which affects transaction consistency.

2. Habitual thinking, the proportion of strong consistency transactions is too high:

Single applications are all local transactions, and strong consistency can be easily guaranteed. After the service, there is no in-depth analysis based on the business scenario, and the strong consistency scheme is habitually adopted, and the business cost is very high.

Experience summary

Based on user experience, taking into account both timeliness and cost

01

In the final consistency, the transaction scheme based on message middleware is adopted.

02

Strong consistency, TCC scheme.

Technical details that can not be ignored-- API openness and Integration

Lesson

1. The internal implementation details are open to the front end / third party:

Includes but is not limited to language-specific implementation details, such as exception classes, inheritance and overloading, abstract interfaces, and so on.

2. Open redundant fields for reuse of back-end data structures:

Sometimes in order to reuse internal data structures and open up the entire object, in fact, consumers only need to use a few of these fields, making the interface less easy to use.

3. Lack of a unified API Gateway:

There is no unified API entrance and precise flow control means.

Experience summary

Improve the ease of use of open API based on API Gateway

Technical details that can not be ignored-- fuse and demotion

Lesson

1. Work behind closed doors with a wide variety of technologies:

Without building a unified micro-service reliability framework, fault injection framework and so on, different business teams fight on their own. Experienced teams, good reliability, poor skills of the team, lack of effective reliability guarantee.

2. Repeated research and development, waste of resources:

Business scenarios are different, and reliability demands are more or less the same. Capabilities such as asynchronous database access, asynchronous Istroke O operations, and micro-service fault isolation have been repeatedly built by different teams.

Experience summary

Integrate mature technologies in the industry to build a unified reliability and fault injection framework

The third-party dependencies are classified and grouped, and the circuit breaker strategy, elegant degradation strategy and timeout strategy are set according to the characteristics of the dependency, in order to achieve differential processing.

Systematic fault scenario carding-- panoramic picture of micro-service fault isolation

Lesson

1. There is no systematic combing of micro-service failure scenarios.

2. Without automated micro-service fault injection and simulation tools, the coverage of micro-service reliability testing scenarios is insufficient, and the problems are left online.

Experience summary

Systematic fault scenario analysis, automatic fault injection and fault isolation measures to improve the reliability of micro-services

Technical details that can not be ignored-- whether microservices must be deployed independently

Lesson

1. Mechanically copy the principle of micro-services, and all micro-services are deployed independently:

Thousands of micro-services, hundreds of nodes cluster network, the number of micro-service processes has expanded to hundreds of thousands, the surrounding supporting facilities can not afford (such as network management OSS).

2. To simplify deployment and management, microservices are all co-located in the same process:

There are some disadvantages, such as loss of upgrade flexibility, inability to scale on demand, poor fault isolation, inability of heterogeneous languages to be integrated, and so on.

Experience summary

The deployment strategy of micro-service is determined according to the split granularity of micro-service, the scale of micro-service, and the maintenance cost acceptable to the operation and maintenance team.

01

Micro-services that can be co-built in a process

A) performance and latency requirements are demanding, and micro-services that require local short circuits can be integrated.

B) A set of microservices that are strongly related to the business for unified management.

C) distributed transactions are not supported for the time being, and local transactions are required to ensure strong consistency of related micro-services (not a long-term solution).

02

Split coarse-grained micro-services, functional independence and cohesion, and it is recommended to deploy independently.

Technical details that can not be ignored-widespread alarm and restlessness

Lesson

1. Alarms flying all over the sky, frightened:

After microservice, the local method invocation becomes remote microservice invocation, and multiple developers will be responsible for multiple microservices of a business process. The alarm has also evolved from a local centralized to a distributed alarm. If the previous alarm method is still used, it will be found that the alarm is flying everywhere, and most of the alarms are repeated and invalid alarms.

Experience summary

Dynamic construction of alarm tree to automatically suppress repeated alarm

01 principle

Combined with the distributed call chain tracking function, the micro-service invocation relationship is dynamically depicted in the running state, and then an alarm tree is constructed. Every time the leaf node is abnormal and needs alarm, it needs to drill down layer by layer along the tree trunk to correlate the alarm and find the alarm Root node. If it can match, it means that the alarm has been made by the root node, and the leaf node gives up the alarm to prevent repeated alarm.

Platform and business relationship-- not Party An and Party B

Lesson

1. Party An and Party B deal with the relationship between micro-service platform and business:

Large-scale business teams, service-oriented requirements are varied, the platform needs to identify high-priority, general-purpose requirements, and most personalized requirements need to be open to the business team for implementation, or planned to follow-up versions. If the business side only starts from the interests of the team, regardless of the overall situation, it is easy to crush the platform, and the final delivery quality can not be guaranteed.

Experience summary

Dare and be good at rejecting unreasonable needs

01 platform:

A) communicate reasonably and argue according to reason.

B) have a deep understanding of the business scenario and grasp the main aspects of the principal contradiction and contradiction.

C) dare to assume responsibility and dare to say "no" to unreasonable needs

02 Business side:

A) continuously improve the practical experience of microservices and be able to distinguish which are platform requirements and which are business requirements.

B) to establish a new relationship of mutual trust and win-win cooperation with the platform.

Pay attention to micro-service technology, pay attention to cloud native, on the official account of "micro-service honeycomb"

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.