Shulou(Shulou.com)06/03 Report--

Problems existing in upgrading continuous service

To achieve continuous service upgrade, you usually need to solve the following problems:

When the service is stopped, it may cause business interruption. During the process of stopping the service, the service may be processing requests, and new requests may be continuously sent to the service.

Under the micro-service architecture, service discovery is generally carried out through the registry, and the client will cache the instance address. When the service is stopped, the consumer may not be aware of the instance offline in time and continue to use the wrong instance for access, resulting in failure.

After the new service is started, there will be a grayscale state and multiple versions coexist. If interfaces are added to the new service, the newly upgraded service needs to correctly send traffic to the service that contains the new interface.

Cost: to achieve continuous service upgrade, you can first prepare a large number of backup machines to start the new service. Then the user's request is drained, and when the old service has no traffic, the old service is stopped. This requires the operation and maintenance staff to prepare additional cluster resources and develop a powerful operation and maintenance monitoring system.

Using the CSE framework, you can easily upgrade continuous service at a very low cost without the help of operation and maintenance tools.

Practice and Evaluation of Rolling upgrade with CSE

When discussing continuous service, you need to design a test and evaluation model first. For simplicity, use the following test scenario to evaluate. The caller accesses application instance 1 and application instance 2 through the gateway and now needs to upgrade the application instance. During the upgrade process, the caller starts N threads to request with the traffic of Mtps. We can evaluate the support of the system for continuous service upgrades by the number of failures in the entire upgrade process. To save resources, we stop version 1.0 of instance 1, then deploy version 2.0 of instance 1, then stop version 1.0 of instance 2, and finally deploy version 2.0 of instance 2. In addition, we also need to construct the server processing delay T to simulate the situation in which the request is being processed.

During this process, the data tested using CSE is as follows:

Number of caller threads N caller traffic M processing delay T failures

10 20tps no 0 10 80tps no 0 10 600tps no 0 10 80tps Implementation principle of 100ms 0

The core mechanisms for realizing continuous service include the following:

Graceful downtime: when the service stops, you need to wait for the request to complete and reject the new request

Retry: for network connection errors and reject request errors, the client needs to select a new server to retry.

Isolation: isolate service instances that have failed more than a certain number of times.

In the above test data, the retry policy is configured as follows:

Servicecomb:

Loadbalance:

RetryEnabled: trueretryOnNext: 1retryOnSame: 0

The quarantine policy is configured to:

Servicecomb:

Loadbalance:

Myservice: isolation: enabled: true enableRequestThreshold: 5 singleTestTime: 10000 continuousFailureThreshold: 2

During testing, when stopping version 1.0 instance 2, you need to ensure that version 2.0 instance 1 has handled the request correctly, otherwise there may be no instances available and upgrade interruptions may occur.

Commercial tools

It is described above to upgrade continuous service only with the help of SDK. By cooperating with CSE's grayscale release and deployment tools, you can achieve a more reliable upgrade and a better experience.

More wonderful articles are on the official micro-service beehive account, come and follow us.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.