Shulou(Shulou.com)06/01 Report--

How to understand the practice of Istio in FreeWheel micro-service, aiming at this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a simpler and easier way.

The pain of FreeWheel's microservices

FreeWheel is a digital video solutions company, our core business system can be understood as a Web ERP. At first, the core business system is completely based on Rails, which is a typical three-tier architecture, but after nearly a decade of development, the maintenance cost of this large single application is getting higher and higher, and it is difficult to expand. In 2016, FreeWheel began to migrate to micro-service architecture, all business functions were rewritten in Golang, and Kubernetes was introduced as a service deployment platform.

However, after migrating to microservices, FreeWheel faces a lot of communication contradictions. On the one hand, the operation and maintenance of the platform layer begins to check with the changes of the application layer. Sometimes upgrade the core network equipment, the whole system will be affected. On the other hand, with the faster and faster iteration of the application layer, the modules begin to contain each other, which not only affects the customer, but also reduces the efficiency of application iteration. It boils down to two problems: one is the control of communication, and the other is the visibility of communication, which can be understood as monitoring.

At first, we tried to use a centralized service gateway Gateway to resolve this contradiction. However, with the migration of more and more modules of micro-services, there are two contradictions: one is that the traffic between services and services influence each other, and the second is that there are many personalized places in the configuration of service gateways in micro-services. the big pot of rice brings complex configuration management, which is becoming more and more difficult to sustain.

In this context, FreeWheel switched to the Service Mesh solution and chose Istio.

The main reason why FreeWheel chose the Istio solution is that FreeWheel's technology stack is Golang and Kubernetes,Istio are currently the most suitable choices.

02

The structure and basic principles of Istio

First, let's take a look at the architecture and fundamentals of Istio. Istio has four core modules:

1. Reverse proxy module: Istio Proxy hijacks all the traffic of Pod and manages the communication in Service Mesh. At the same time, the traffic that manages the interaction between the internal and external boundaries of the Mesh is also a reverse proxy.

2. Pilot module: manages the dynamic configuration of all Service Mesh.

3. Citadel module: mainly to automatically maintain the certificate of communication between services.

4. Mixer module: two sets of services are deployed in Kubernetes: one set of services provides authorization and capacity checking capabilities, and the other group of Policy provides data collection capabilities, through which the data are summarized.

If you look at the design of the whole system, it can also be divided into four parts:

1. Reverse proxy.

two。 At present, network security is mainly compatible with Spiffe standard.

3. Dynamic configuration management of Proxy access to K8s for C++.

4. For the Attribute finite state machine model, authorization, capacity, management monitoring, and so on all the foundation.

03

Micro-services managed by Istio

What happens after Service Mesh deploys pod?

1. Before creating Pod, Sidecar Injector first adds custom initContainer, sidecar container and volume to Pod, where volume is related to the communication security of Istio, and Istio maintains a dynamic authentication key for Pod.

two。 Next, create a container to start Pod, where initContainer first generates iptables rules for traffic hijacking for the current Pod.

3. Then start the sidecar and the actual application container in Pod.

4. Next, sidecar and Pilot establish a connection to accept dynamic configuration and updates; establish a connection with Mixer (policy) to check authorization, capacity, etc.; and establish a connection with Mixer (telemetry) to report traffic-related monitoring metadata.

After Pod is started and connected to Service Mesh, there are two more components:

Galley: implements the verification of dynamic configuration.

Citadel: check the validity of mount's secret in Pod and automatically distribute legal certificates for Pod.

04

How does FreeWheel make full use of Istio?

FreeWheel already has a complex custom authentication and authorization mechanism. In order to make full use of Istio, we integrate these systems by extending Istio, which involves two aspects:

Extended Sidecar: add authentication support, provide authentication support for business systems, and transfer user-related information to Mesh in the form of header. Subsequent authorization, monitoring and current restriction can be completed with the native mechanism of Istio.

Extended Mixer: select a portion of the traffic to apply the corresponding authorization logic.

Extended Sidecar access authentication

FreeWheel already has a simple service gateway implementation, which extracts the authentication logic into this module, and inserts some header for the downstream services, such as the user information obtained after authentication, in the reverse proxy after authentication.

Istio does not develop such an interface that is fully customized to requests, so in order to modify the contents of requests and responses, you must define reverse proxies. Instead of replacing Envoy, we implement it with a reverse proxy, which is also connected to the configuration management service, which can use the native K8s configuration management interface to achieve dynamic configuration management. However, it should be noted that the traffic of containers in Sidecar is not hijacked by Iptables by default. If you need to include traffic hijacking, you need to explicitly specify Aannotation to include ports or CIDR.

To connect to Sidecar is to modify the ConfigMap of Istio-system/istio-sidecar-injector and add a custom reverse proxy.

This reverse proxy is very simple and overwrites the mapping relationship between token and user information. If token is passed into the upstream, the user information will be crammed into the header and passed downstream, and the traffic will be forwarded to the application service. There is also a process of checking authorization and capacity between the reverse proxy and the application service, which is achieved by customizing the Mixer.

By adding FreeWheel custom authentication support in Sidecar, downstream can make full use of the authorization, current limit and monitoring interfaces provided by Istio. However, after using Sidecar Injector for some time, I found that there were two problems:

Sidecar does not have the secret automatically injected by K8s, nor can it automatically establish a Master connection through the environment variables in the container, so additional Kubernetes needs to be managed.

Service traffic in Sidecar is not hijacked by default. Additional Annotation needs to be added if hijacking is needed.

Extended Mixer access authorization

When we implement highly customized request and response content, we use Sidecar to implement the extension, but when the service authorizes and customizes the current limit, many similar features do not need to modify the request response, but just calculate before the request reaches the downstream service, and the result is rejected or passed. At this point, it can be extended through Mixer.

Mixer is a highly customizable state machine model, and Envoy provides two interfaces: Report and Check. These two interfaces are called many times later in the life of the connection and request, and functions such as Quota,Metrics,Tracing that we often use are implemented through it.

The following figure shows the basic principles of Mixer. Template is a framework for specific processing mechanisms for Attribute reported by Proxy, supporting four categories:

Preprocess: summarizes traffic-related metadata and environment (K8s)-related metadata.

Report: report data.

Check: decides whether or not to allow current access.

Quota: decide whether the capacity is sufficient.

The basic principle of Mixer

How does FreeWheel extend Mixer?

First, an adapter is added to implement the authorization template. In fact, FreeWheel already has a default RBAC template. But when we add a new authorization capability, we want to be absolutely controllable. For example, through a dynamic configuration, turn on the authorization when there is only a portion of the user or access to a certain type of URL, and turn it off at other times.

Mixer provides a very flexible model, which allows Handler to dynamically select a part of the traffic to introduce additional mechanisms (such as permission control, current restriction, etc.). This is a very important capability in application operation and maintenance. As long as the request and response functions are not modified, extended Mixer can be used.

Here, mymock completely rejects all matched traffic:

The basic principle of mymock Handler

There are three key elements in extending Mixer:

1. Configuration of Handler. This is a default behavior that allows checknothing to return to 400 when matching to user1, and then reject them all.

two。 System process instance. This is the fixed thing of each module, after defining this kind of input, there can be a black box, which is an object-oriented concept.

3. RUL . When the user identity is user1, the instance will be processed by handler. If you want to expand, the first step is to define the data structure of handler; the second step is to implement the interface of time, where there are two interfaces; and the third is to generate the interface of the relevant go in the defined handler file through code. Finally, register into mixer, recompile and package mixer, and you can use it directly.

It should be noted that mixer accesses an authorized policy module, which may affect the inactivity of the service grid (Service Mesh). Therefore, when redeploying these modules, if you can't stand a short downtime, you may need to do some grayscale publishing mechanism.

For Istio configuration management, we also found some performance bottlenecks and limitations.

There are performance bottlenecks in Kubernetes and configuration management for etcd. A single Resource is controlled at the K level and can work normally. Once it reaches the ten thousand level, it may not work, such as tens of thousands of single Resource.

Limitations of the Istio configuration itself. The first is anti-jitter processing. No matter how fast the deployment in the cluster changes, it will not block Istio. Second, Istio other configurations do not have anti-jitter processing, once the program is plugged in, be sure to do current limit. In the end, Istio uses a compatibility design that CRD cannot do, so there is no way to upgrade CRD smoothly.

What will FreeWheel do in the future?

The first is Service Contract, which encapsulates the complexity of Istio and other configurations in the platform layer, and abstracts a secure and efficient application operation and maintenance system, leaving room for future technical improvements.

Then there is Chaos Testing, which solves the problems of continuous operation and risk control of complex micro-service systems. This is also a popular theme at present.

On-site question and answer

1. For the entry of external traffic into Service Mesh, when there is an Ingress in Kubernetes and a Gateway in the application of micro-service Service Mesh, it transfers Gateway to Service Mesh. How can these three be linked so that external traffic can enter smoothly?

A: it's very simple. Service Mesh can be understood as a closed box. How did all your external Ingress work before and how did it work? let's talk about how it works in Service Mesh. If you think of it as a black box, now you have an Ingress, you can transfer it to the Ingress Gateway in the Istio. In the past, the traffic will be forwarded directly to each Pod, directly to the Pod, from the Pod to the Sidecar, and then to the application.

two。 The Sidecar you just mentioned should directly block the traffic in the Pod. Only through the Gateway traffic in the Service Mesh can you enter the Pod?

A: no. As a matter of fact, there are no restrictions on traffic forwarding. Every reverse proxy in Service Mesh can assume the function of Ingress Gateway. It is only a matter of routing configuration. If you get this right, you can configure this feature on any Service Mesh. The reverse proxy is identical.

3. I use the Ingress in Istio's Gateway,Kubernetes to turn off and be able to communicate with the outside world.

A: this is the mode recommended by Istio. When you manage the incoming traffic outside the cluster, you can directly go to Ingress Gateway, which can be well limited.

4. Did you encounter any pits during your landing?

A: yes. At first, we tried to transfer the permission data of the business directly to the native data structure, because we wanted to use it for permission control. But then it turns out that it can't control such a large amount of data. The second pit is that Istio does not have anti-jitter treatment for some of its Resource, which may cause the Service Mesh behavior to be very unstable, so current restriction must be done on the client side.

In addition, in the long run, these modules of Istio are still a single point, which may cause Service Mesh to be unavailable. We have encountered inexplicable problems after running for three months before, and we have to do exercises frequently to maintain this kind of monitoring system. Our company now conducts exercises once a month, regularly brings out the problems in a main production environment, and then checks whether each link is working properly, which is also a thing that we will develop in the future.

This is the answer to the question on how to understand the practice of Istio in FreeWheel microservice. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel for more related knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.