Network Security Internet Technology Development Database Servers Mobile Phone Android Software Apple Software Computer Software News IT Information

In addition to Weibo, there is also WeChat

Please pay attention

WeChat public account

Shulou

How to use Istio fuse

2025-01-15 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Servers >

Share

Shulou(Shulou.com)05/31 Report--

This article mainly explains "how to use Istio fuses". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn how to use Istio fuses.

Preface

Istio is praised for its flexible observability and secure inter-service communication. However, other more important functions really make Istio a Swiss Army knife in the service grid. When it comes to SLO problems such as running time, delay and error rate, the traffic management ability between services is very important.

When Istio operator was released earlier this year, our goal (in addition to managing the installation and upgrade of Istio) was to support these excellent traffic routing features while making all features easier to use. Finally, we created a simple and automated service grid Backyards that provides the ability to manage UI, CLI, and GraphQL API on top of Istio operator. Backyards is integrated into Banzai Cloud's container management platform Pipeline and can also work independently as a single product. Of course, using Backyards with Pipeline provides special benefits for users (such as managing applications in multi-cloud and hybrid cloud environments), and Backyards can also be used in any Kubernetes installation environment.

Circuit breaker: failure is an option

In a micro-service architecture, services may be implemented in different languages and deployed on multiple nodes or clusters with different response times or failure rates. If the service responds to the request successfully (and in a timely manner), its performance is satisfactory. However, this is not the case, and downstream clients should be protected when upstream services are too slow. Conversely, upstream services must also be protected from being dragged down by the backlog of requests. In the case of multiple clients, the situation can be more complex and may lead to a series of cascading failures of the entire infrastructure. The solution to this problem is to use a time-tested fuse mode.

A fuse can have three states: off, open, and half-open, which is off by default. In the off state, regardless of whether the request succeeds or fails, the circuit breaker will not be triggered until the preset threshold of the number of failures is reached. When the threshold is reached, the fuse is turned on. When a service in the open state is invoked, the fuse disconnects the request, which means that it returns an error directly without executing the call. Cascading failures can be prevented in the production environment by disconnecting downstream requests on the client. After a pre-configured timeout, the fuse enters a half-open state, in which the fault service has time to recover from its interrupted behavior. If the request continues to fail in this state, the fuse will open again and continue to block the request. Otherwise, the fuse will be turned off and the service will be allowed to process the request again.

Fuse in Istio

The circuit breaker of Istio can be configured in the flow policy. In Istio's custom resource Destination Rule, there are two circuit breaker-related configurations under the TrafficPolicy field: ConnectionPoolSettings and OutlierDetection.

ConnectionPoolSettings can configure the number of connections for the service. OutlierDetection is used to control the removal of unhealthy instances from the load balancer pool.

For example, ConnectionPoolSettings controls the maximum number of requests, suspending requests, retrying, or timeout; the number of requests that have errors when the OutlierDetection setting service is removed from the connection pool, you can set the minimum eviction time and the maximum eviction percentage. For a complete list of fields, please refer to the documentation.

Istio uses the fuse feature of Envoy at the bottom.

Let's take a look at the configuration of circuit breakers in Destination Rule:

ApiVersion: networking.istio.io/v1alpha3kind: DestinationRulemetadata: name: notificationsspec: host: notifications trafficPolicy: connectionPool: tcp: maxConnections: 1 http: http1MaxPendingRequests: 1 maxRequestsPerConnection: 1 outlierDetection: consecutiveErrors: 1 interval: 1s baseEjectionTime: 3m maxEjectionPercent: 100

Using these settings in the ConnectionPoolSettings field, only one connection can be established with the notifications service at a given time: there can be at most one pending request per connection. If the threshold is reached, the fuse will begin to block the request.

The settings in the OutlierDetection section are used to check whether there is an error in invoking the service per second. If so, remove the service from the load balancer pool for at least three minutes (the maximum pop-up percentage of 100% means that all service instances can be ejected at the same time if necessary).

One thing to pay special attention to when manually creating Destination Rule resources is whether mTLS is enabled for the service. If so, you also need to set the following fields in the Destination Rule, otherwise the caller may receive a 503 error when calling the movies service:

TrafficPolicy: tls: mode: ISTIO_MUTUAL

You can also enable global mTLS for a specific namespace or service. You should know these settings to determine whether to set trafficPolicy.tls.mode to ISTIO_MUTUAL. More importantly, when you try to configure a completely different function (such as a fuse), it's easy to forget to set this field.

Tip: always consider mTLS before creating a Destination Rule!

To trigger the circuit breaker, let's invoke the notifications service from both connections at the same time. The maxConnections field is set to 1. You should see 503 and 200 arriving at the same time.

When a service receives more load from the client than it can handle (as configured in the fuse), it returns a 503 error before calling. This is one way to prevent error cascading.

Monitor and control

Your service must be monitored in a production environment so that you can be notified and be able to check if an error occurs in the system. So, if you have configured a fuse for your service, you will want to know when it tripped, how many percent of the requests were intercepted by the fuse, when it was triggered, and from which downstream client? If you can answer these questions, you can determine if the fuse is working properly, fine-tune the configuration as needed, or optimize the service to handle additional concurrent requests.

Tip: if you keep reading, you can see and configure all of these settings in Backyards UI.

Let's see how to determine the fuse trip in Istio:

The response code when the fuse is tripped is 503, so you cannot distinguish it from other 503 errors just by that response. In Envoy, there is a counter called upstream_rq_pending_overflow, which records the total number of requests that have been cut and failed. You can get this information if you delve deeper into Envoy statistics for your service, but it's not easy.

In addition to the response code, Envoy returns a response flag, and there is a dedicated response flag to indicate fuse trip: UO. This will not be particularly useful if this flag is available only from Envoy logs. Fortunately, it is implemented in Istio, so the response flag is available in the Istio metric and can be obtained by Prometheus.

The trip of the fuse can be queried like this:

Sum (istio_requests_total {response_code= "503", response_flags= "UO"}) by (source_workload, destination_workload, response_code) Backyards is easier to fuse.

When using Backyards, you do not need to manually edit the Destination Rules to set the fuse. The same result can be achieved through a convenient UI interface or (if you prefer) a Backyards CLI command line tool.

Don't worry about mismatching the Destination Rules because you forgot to set trafficPolicy.tls.mode to ISTIO_MUTUAL. Backyards will solve this problem for you; it will find the service with mTLS enabled and set the above fields accordingly.

The above is just an example of the Backyards validation feature, which can avoid your setting errors. There are more features to follow.

On top of this, you can see the visual interface and activity dashboard of services and requests, so you can easily determine how many requests are triggered by fuses, which caller it comes from and when.

Fuse actual combat to create a cluster

First, we need a Kubernetes cluster.

I created a Kubernetes cluster on GKE through the free development version of Pipeline platform. If you want to do the same, you can create clusters locally at the five cloud providers we support or using Pipeline. Otherwise, you need to provide your own Kubernetes cluster.

Install BACKYARDS

The easiest way to install Istio,Backyards and demo applications in a new cluster is to use Backyards CLI.

All you need to do is execute the following command (the cluster must have KUBECONFIG set):

$backyards install-a-run-demo

This command first installs Istio using our open source Istio operator, and then installs the Backyards and demo applications. After the installation is complete, Backyards UI will automatically open and send some traffic to the demo application. With this simple command, you can see that Backyards has started a new Istio cluster in a few minutes! Try it!

You can also perform all these steps in order. Backyards requires an Istio cluster-- if not, it can be installed via $backyards istio install. Once you have installed Istio, you can install Backyards using $backyards install. Finally, deploy the demo application using $backyards demoapp install.

Tip: Backyards is the core component of the Pipeline platform-you can try the developer version (Service Mesh tab).

Create a fuse configuration fuse using BACKYARDS UI

You do not need to manually create or edit the Destination Rule, you can easily change the configuration of the fuse in the UI interface. Let's create a demo first.

As you will see, Backyards (compared to Kiali) is not only a web UI built for observability, but also a feature-rich service grid management tool that supports both single and multi-clusters, and has powerful CLI and GraphQL API.

View fuse settin

You don't need to view the fuse configuration through Destination Rule (for example, through kubectl), but when you click the notification service icon and switch the SHOW CONFIGS slider, you can see them on the right side of the Backyards UI.

Monitoring fuse

According to the previous setting, when two connections generate traffic at the same time, the fuse will issue a trip request. In Backyards UI, you will see that the edges of the graph appear red. If you click on the service, you will learn more about the error and see two real-time Grafana dashboards dedicated to displaying fuse tripping.

The first dashboard shows the percentage of total requests triggered by fuses. When there is no fuse error and your service is working properly, this picture will show 0%. Otherwise, you will be able to see immediately how many requests are triggered by fuses.

The second dashboard provides tripping failures caused by source fuses. If no tripping occurs, there will be no spike in this figure. Otherwise, you will see which service caused the tripping, when tripping, and the number of tripping times. You can use this figure to track malicious clients.

These are real-time Grafana dashboards that display circuit breaker related information. By default, Backyards integrates Grafana and Prometheus--, and there are more dashboards that can help you drill down into service metrics.

Remove fuse configuration

The fuse configuration can be easily removed through the Remove button.

The actual battle of Backyards UI circuit breaker

This video summarizes all these UI operations.

Create a fuse using BACKYARDS-CLI

From experience, what can be done from the UI interface can also be done through the Backyards CLI command line tool.

Configuration fuse

Let's do the test of creating a circuit breaker again, this time through the CLI command line.

This can be done in interactive mode:

$backyards r cb set backyards-demo/notifications? Maximum number of HTTP1/TCP connections 1? TCP connection timeout 3s? Maximum number of pending HTTP requests 1? Maximum number of requests 1024? Maximum number of requests per connection 1? Maximum number of retries 1024? Number of errors before a host is ejected 1? Time interval between ejection sweep analysis 1s? Minimum ejection duration 3m? Maximum ejection percentage 100INFO [0043] circuit breaker rules successfully applied to 'backyards-demo/notifications'Connections Timeout Pending Requests Requests RPC Retries Errors Interval Ejection time percentage1 3s 1 1024 1 1024 1 1s 3m 100

Or, in non-interactive mode, specify the value to set:

Backyards r cb set backyards-demo/notifications-- non-interactive-- max-connections=1-- max-pending-requests=1-- max-requests-per-connection=1-- consecutiveErrors=1-- interval=1s-- baseEjectionTime=3m-- maxEjectionPercent=100Connections Timeout Pending Requests Requests RPC Retries Errors Interval Ejection time percentage1 3s 1 1024 1 1024 5 1s 3m 100

After the command is executed, the circuit breaker configuration is obtained and displayed immediately.

View circuit breaker settin

You can use the following command to list the settings of the fuse through namespace:

$backyards r cb get backyards-demo/notifications Connections Timeout Pending Requests Requests RPC Retries Errors Interval Ejection time percentage 1 3s 1 1024 1 1024 5 1s 3m 100s

By default, the results are displayed in tabular form, and JSON or YMAL format is also supported:

$backyards r cb get backyards-demo/notifications-o json {"maxConnections": 1, "connectTimeout": "3s", "http1MaxPendingRequests": 1, "http2MaxRequests": 1024, "maxRequestsPerConnection": 1, "maxRetries": 1024, "consecutiveErrors": 5, "interval": "1s", "baseEjectionTime": "3m" "maxEjectionPercent": 100} $backyards r cb get backyards-demo/notifications-o yaml maxConnections: 1 connectTimeout: 3s http1MaxPendingRequests: 1 http2MaxRequests: 1024 maxRequestsPerConnection: 1 maxRetries: 1024 consecutiveErrors: 5 interval: 1s baseEjectionTime: 3m maxEjectionPercent: 100Monitoring fuse

To view a dashboard similar to the previous Grafana UI interface from CLI, you can trigger the trip by invoking the service from multiple connections and execute the command:

$backyards r cb graph backyards-demo/notifications

You can see results similar to the following:

Remove fuse configuration

Remove the fuse and execute the following command:

$backyards r cb delete backyards-demo/notificationsINFO [0000] current settingsConnections Timeout Pending Requests Requests RPC Retries Errors Interval Ejection time percentage1 3s 1 1024 1 1024 5 1s 3m 100? Do you want to DELETE the circuit breaker rules? YesINFO [0008] circuit breaker rules set to backyards-demo/notifications successfully deleted

Verify success using the following command:

$backyards r cb get backyards-demo/notifications INFO [0001] no circuit breaker rules set for backyards-demo/notifications creates a fuse using BACKYARDS GRAPHQL API

Backyards consists of multiple components, such as Istio, Banzai Cloud's Istio operator, multi-cluster Canary release operator, and multiple back-end infrastructure. All of this is behind the Backyards' GraphQL API.

Both Backyards UI and CLI use Backyards's GraphQL API, which will be released with the GA version at the end of September. Users will soon be able to use our tools to manage Istio and build their own clients.

Clear

Remove the demo application, Backyards, and Istio from your cluster, execute the following command, and it will uninstall these components sequentially:

Backyards uninstall-a Thank you for your reading, the above is the content of "how to use Istio fuses", after the study of this article, I believe you have a deeper understanding of how to use Istio fuses, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.

Views: 0

*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.

Share To

Servers

Wechat

© 2024 shulou.com SLNews company. All rights reserved.

12
Report