Shulou(Shulou.com)06/02 Report--

Cascading failure scenario of circuit breaker background micro-service

In a distributed environment, microservices call each other. In some cases, such as back-end middleware service failure or third-party service interruption, a service is unavailable indefinitely and cannot be recovered for a short time, which may lead to cascading failures and eventually overwhelm the entire business cluster.

Circuit breaker and retry

The circuit breaker mode is different from the retry mode, which enables the application to retry the operation in the expectation that it will succeed, while the circuit breaker mode prevents the application from performing a possible failed operation and reduces the waste of CPU, memory, threads and other resources that may fail, thus ensuring the overall availability of the service.

Circuit Breaker Design Analysis of Circuit Breaker based on Agent Mode

A circuit breaker is equivalent to an agent requesting the execution of an operation, hosting the execution of the requested operation

Realize the principle flow:

Intercept the request executed by the service, decide whether to return directly through the current status, if otherwise perform subsequent operations to attempt to perform the operation, and obtain the return result, determine the status of the current circuit breaker according to the returned result and current statistical information, modify the status to return the execution result circuit breaker state machine

There are three states in the implementation of circuit breaker state machine: Closed (circuit breaker off), Open (open), and HalfOpen (semi-open)

Status description remarks Closed close circuit breaker close normal operation Open open circuit breaker open, all requests directly return errors, do not execute any requests HalfOpen semi-open allows a limited number of requests to pass, if the execution is successful, return to the closed state, if still fails, return to open, and then restart the timeout timer

# implementation of circuit breaker

Realization principle diagram

The implementation of circuit breaker is mainly divided into three parts: state statistics, state transfer, request execution.

Status statistics: count the number of successful and failed requests that have been executed to determine whether a state transition is required

State transition: determine and transfer the target state according to the current statistical information and current state

Request execution: acting for the execution of front-end tasks. If there is no attempt to execute the current state, an error will be returned directly to avoid waste of resources.

There is an open source implementation in Golang. Next, https://github.com/sony/gobreaker/blob/, will analyze its implementation.

Status Statistics-counter Counts

Counts is a counter that records the number of successes and failures of current requests

Type Counts struct {Requests uint32 / / number of requests TotalSuccesses uint32 / / successful TotalFailures uint32 / / failed ConsecutiveSuccesses uint32 / / continuous successful ConsecutiveFailures uint32 / / continuous failed}

The counter completes the number of times the corresponding request status is completed, and provides data for the subsequent state transfer. Counts provides several auxiliary interfaces such as onRequest, onSuccess, onFailure and clear to implement the operation of the corresponding request status. If you are interested, please see

State machine-CircuitBreakertype CircuitBreaker struct {name string / / maxRequests limits the maximum number of requests in the half-open state to avoid massive requests to use the service failure in the recovery process maxRequests uint32 / / interval for how often the circuit breaker clears the Counts information in the closed state. If set to 0, the Counts interval time.Duration / / timeout will not be cleared in the closed state to enter the open state. How long does it take to switch to the half-open state? the default is 60s timeout time.Duration / / readyToTrip circuit breaker condition. When the execution fails, it will decide whether to enter the Open state readyToTrip func (counts Counts) bool / / onStateChange circuit breaker status according to the readyToTrip to change the callback function onStateChange func (name string, from State, to State) mutex sync.Mutex / /. State circuit breaker status state State / / generation is a progressive increment, which is equivalent to the number of current circuit breaker state switches. In order to avoid the statistical impact of uncompleted requests on new states after state switching, if a requested generation is found to be different from the current generation The statistical count generation uint64 / / Counts statistics counts Counts / / expiry timeout expiration is used to switch from open state to half-open state. When the timeout occurs, it will switch from open state to half-open state expiry time.Time} core process CircuitBreaker.Execute

Request execution, request execution interface that is open to the outside world

Func (cb * CircuitBreaker) Execute (req func () (interface {}, error)) (interface {}, error) {/ / execute the request hook according to the current status To return the current generation and err (not nil if located in open and half-open), and determine whether to return generation directly by err, err: = cb.beforeRequest () if err! = nil {return nil, err} / / capture panic Avoid circuit breaker panic defer func () {e: = recover () if e! = nil {cb.afterRequest (generation, false) panic (e)}} () / / execute request result, err: = req () / / calculate the corresponding state according to the result, and pass generation cb.afterRequest (generation) at the same time Err = = nil) return result, err} CircuitBreaker.beforeRequestfunc (cb * CircuitBreaker) beforeRequest () (uint64, error) {cb.mutex.Lock () defer cb.mutex.Unlock () / / get the current status now: = time.Now () state, generation: = cb.currentState (now) / / open and half-open status is returned directly if state = = StateOpen {return generation ErrOpenState} else if state = = StateHalfOpen & & cb.counts.Requests > = cb.maxRequests {/ / avoid the impact of massive requests on the service being restored There is a current-limiting operation. Avoid requests exceeding the maximum number of requests return generation, ErrTooManyRequests} / / Statistical status cb.counts.onRequest () return generation, nil} CircuitBreaker.afterRequestfunc (cb * CircuitBreaker) afterRequest (before uint64, success bool) {cb.mutex.Lock () defer cb.mutex.Unlock () / / retrieve status now: = time.Now () state Generation: = cb.currentState (now) / / if the state is inconsistent Do not count if generation! = before {return} / / based on the state count if success {cb.onSuccess (state, now)} else {cb.onFailure (state, now)}} CircuitBreaker.currentStatefunc (cb * CircuitBreaker) currentState (now time.Time) (State, uint64) {switch cb.state {case StateClosed: / / if the current closed status And if expiry is set, increment Generation to the new round of statistical count if! cb.expiry.IsZero () & & cb.expiry.Before (now) {cb.toNewGeneration (now)} case StateOpen: / / if it is Open status and times out Try to half open if cb.expiry.Before (now) {cb.setState (StateHalfOpen, now)} return cb.state, cb.generation} CircuitBreaker.toNewgenerationfunc (cb * CircuitBreaker) toNewGeneration (now time.Time) {/ / incremental generation Clear status cb.generation++ cb.counts.clear () / / set timeout var zero time.Time switch cb.state {case StateClosed: if cb.interval = = 0 {cb.expiry = zero} else {cb.expiry = now.Add (cb.interval)} case StateOpen: cb.expiry = now.Add (cb. Timeout) default: / / StateHalfOpen cb.expiry = zero}} summarize the golden link of the circuit breaker

BeforeRequest: whether the current request can be executed after the current request is completed, and the status timeout switch, while returning the current genenrationreq: execute the request afterRequest: complete the status statistics of the request, and determine the advantages and disadvantages of the state switching circuit breaker

The circuit breaker is more suitable for the invocation of remote services or third-party services. If the operation is very likely to fail, the circuit breaker can minimize the impact of failure on the application and avoid waste of resources.

But the disadvantage is also obvious, the circuit breaker itself is equivalent to a layer of agents, statistics and control in the application execution, itself has a certain resource consumption, while the internal implementation based on synx.Mutex locks, there must be lock contention problems under high concurrency, it may be necessary to use multiple circuit breakers according to the business to disperse this kind of lock contention, and should be avoided in the circuit breaker req function To perform retries and long timeout waits, because the core of the circuit breaker is fast failure

More articles can be found at http://www.sreguide.com/

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.