Shulou(Shulou.com)06/02 Report--

Author | Cao Shengli Apache Dubbo PMC

Guide: the stereotype of Dubbo as a high-performance Java RPC framework has long been deeply rooted in the hearts of the people, in the architecture selection of Cloud Native, Spring Cloud may be the priority choice of the industry. In fact, Dubbo has quietly evolved into Cloud Native infrastructure, not only inheriting the glory of the past RPC era, but also improving the lack of existing infrastructure. Since the container and K8s appeared on the stage, it has brought great challenges to the original field of RPC. This paper mainly describes the problems encountered in the field of RPC and some thoughts on how RPC embraces K8s.

Introduction to K8s

Kubernetes is an open source application used to manage containerized applications on multiple hosts in the cloud platform. The goal of Kubernetes is to make the deployment of containerized applications simple and efficient. Kubernetes provides a mechanism for application deployment, planning, updating and maintenance. Kubernetes is abbreviated to K8s.

Cdn.com/90e5b6bb184faa3ce95c9494ae165a8d1841dce9.jpeg ">

In Kubernetes, the smallest administrative element is not individual containers, but Pod. The life cycle of Pod needs to be aware of the following:

Containers and applications can be killed at any time; Pod Ip and hostnames may change (unless customized using StatefulSet); files written to local disks may disappear, and storage volumes are needed if you want to avoid invalidation. The relational application of Application & Container & Pod is deployed in a container. In general, an application can only be deployed in one container; a Pod can contain one or more containers. In general, only one container is recommended for a Pod. Except for the following scenarios: side car; one container has been running since it was run with another local container. For example, an application under one container is responsible for downloading data, while an application under another container provides services. Service

If some Pods provides some features for other Pod to use, how is it realized in the Kubernete cluster that these foreground can be tracked continuously to these backgrounds? The answer is Service.

Kubernete Service is a policy abstraction that defines a set of Pod, and the Pod marked by the service is generally determined by label Selector. Service abstracts access to Pod.

The default Service, A Record is obtained through a cluster IP. But sometimes you need to return a list of all the Pod IP that meet the criteria, and you can use Headless Services directly.

Reference: https://kubernetes.io/

Recommended books: [kubernetes in action]

Introduction and Analysis of RPC

With the popularity of micro-services, there are enough mature solutions for communication between applications.

After Dubbo was open source in 2011, it was adopted by a large number of small and medium-sized companies; after the launch of Spring Boot, Spring gradually glowed into the second spring, and then Spring Cloud came out, gradually occupying the market, and fought with Dubbo Chamber in the Chinese market; gRPC is an end-to-end communication tool based on Http2 launched by Google, which gradually occupies a dominant position in the K8s market. For example, etcd,Istio uses gRPC as a communication tool. Service Mesh has been hot conceptually from the beginning, and now it is gradually becoming mature; Istio + Envoy (other sidecar) is gradually beginning to take the stage. Application developer perspective

From a functional perspective, the features that are perceived to developers are:

Service implementation service exposure (annotation or configuration) service invocation (annotation or configuration) service governance, etc.

From the point of view of selection, we will pay attention to the following points:

The perspective of framework developers such as ease of use (development ease of use and out-of-the-box) performance and functional scalability

Key process:

Service exposure Service Registration Service Discovery Service invocation Service Governance

Key knowledge points:

Serialization of network communication service routing load balancing service current limiting fuse degradation and other service governance mainstream technologies to implement Dubbo / HSF

Dubbo provides interface-oriented remote method calls. Application developers define interfaces, write services and expose them; the Client side invokes through interfaces; the dimension of Dubbo registration service is the interface dimension, and each interface writes a piece of data in the registry; Dubbo supports conditional routing, script routing, Tag routing, etc. These routing rules are strongly dependent on IP addresses.

Note: most of the mechanisms of Dubbo and HSF are similar, so Dubbo is discussed below.

SpringCloud

Spring Cloud makes network calls in the form of Rest. Application developers can write their own exposed Rest services, such as springmvc.

The service registration in Spring Cloud is the application dimension (Eureka), and the Client side and the Server side communicate by agreement.

Spring Cloud provides a set of standard API, of which Netflix is the best. This set of API is implemented. For most developers, they can directly rely on and use Netflix, so it can be said that Netflix has become the core of Spring Cloud. However, as a commercial company, the investment in open source is often changeable, such as the institutional maintenance of Eureka.

GRPC

GRPC is a RPC framework based on HTTP/2 protocol, which uses Protobuf as IDL. As an end-to-end communication scheme, gRPC can solve the current multilingual problem.

GRPC itself does not provide the functions of service registration and service governance. But now we can see that gRpc has a tendency to expand its ambitions in this area.

K8s

There is no fair communication framework in K8s system, so gRPC is generally recommended as RPC framework.

By default in the K8s system, the IP of Pod changes, so if communication is needed between Pod and Pod, there are several ways:

Service+DNS: create a new Service. You can select a set of Pod lists through the tag. This Service corresponds to an unchanged cluster IP. The Client end accesses the cluster IP via DNS or directly. This cluster IP is roughly equivalent to the implementation of load balancing (iptable mode). The difference between headless service:headless service and the above service is that it does not provide cluster IP and obtains a set of IP lists in the form of hostname, and the Client side decides which Pod to access. Istio + Envoy

The control layer of Istio will request and listen for pod information, service information and other information from the Api server of K8s. In this way, you have complete information about the pod,service and so on in the complete K8s cluster in Istio. If there is a change in the information in the K8s cluster, you can also be notified in Istio and update the corresponding information.

Envoy, as one of the most common implementations of Proxy, is briefly introduced with Envoy as an example. Envoy dynamically discovers resources by querying files or managing servers. The corresponding discovery service and its corresponding Api are called xDS. The content of the agreement includes LDS, RDS, CDS and so on.

Reference:

Introduction to Service Mesh: https://www.infoq.cn/article/pattern-service-mesh[Istio]

Routing rules: https://istio.io/docs/tasks/traffic-management/request-routing/

Remarks: the above knowledge is obtained through consulting materials (Istio official website) and communicating with Service Mesh students of the group. If you have any questions, please correct them.

Summary

Problems and challenges encountered the symbiosis of Spring Cloud and Dubbo

Dubbo is based on TCP communication by default, and Spring Cloud is mostly based on Rest requests. In the process of commercialization of Aliyun, it is found that a large number of companies need Spring Cloud applications to communicate with Dubbo, and the community mainly relies on adding a layer of gateway on Dubbo.

Is there a solution for unified service registration discovery and service invocation?

The basic theory can be referred to:

Https://yq.aliyun.com/articles/585461

The Challenge of Dubbo in K8s scenario

The IP of Pod under K8s is variable (default), and the service governance of Dubbo is highly dependent on IP.

The service registration of K8s is done through the definition of Pod, and service discovery is actually the process of finding Pod. There is a corresponding relationship between Pod and the application, which does not match the service registration discovery model of the interface dimension in Dubbo.

The living Space of Dubbo in Service Mesh scene

Dubbo needs support tailoring, and most of the functions of Dubbo can be done by sidecar (proxy).

If the company is already deploying the RPC framework, is there a good transition plan if you need to implement Service Mesh at this time?

Problem carding service definition

How do you define a service? You need to look at how to define services from the perspective of application developers.

The service corresponds to a function in the functional dimension, such as querying the details of purchased orders. In Dubbo, it corresponds to a method under an interface; in Spring Cloud and gRPC, there is a http request.

From a function-oriented programming perspective, a service is a function. In the Java language, class is the basis of all programming, so some services are aggregated according to a certain dimension and put into an interface, which becomes an interface containing a lot of services.

From the perspective of Dubbo, Dubbo is a remote communication oriented to interface programming, so Dubbo is a set of service-oriented programming. if you want to call a service, you must introduce it through the interface, and then call a service in the interface. In fact, the service query function provided in Dubbo Ops is not to query a single service, but to obtain a specific method (service) through the query interface (service set).

In the Spring Cloud world, the service provider registers its application information (Ip+port) as an instance under the application, and the service consumer and the service provider make Rest requests in the agreed form. Each request also corresponds to a service.

The difference between Service in K8s and K8s

The Service in K8s actually corresponds to a set of pod+port lists and is closely related to DNS; it is expressed in an easy-to-understand way: maintains the relational mapping of the pod set. And the services mentioned above are two concepts that belong to different scenarios.

Services are defined in this way, and the granularity of service governance is also based on service granularity. You can set timeouts for each service, set routing rules, and so on. But what does the granularity of service registration have to do with services?

Service registration granularity

An application contains many interfaces, an interface contains many services (Dubbo), or an application contains many services (Spring Cloud). Analyze the advantages and disadvantages of application dimension registration and interface dimension registration. There will be a separate article on the application of dimensional registration.

Interface dimension registration

Advantages:

It is very convenient to query the service query according to the interface dimension, and it is not difficult to implement. When the application is split or merged, the Client end (consumer) does not need to care, so that the user has no feeling.

Disadvantages:

The corresponding relationship between the model and the mainstream platforms such as K8s does not match; the amount of registered data is very large, which has a certain performance risk. Application dimension

Advantages:

The corresponding relationship is consistent with the models such as K8sjingle Spring Cloud, etc., and the performance can be greatly alleviated.

Disadvantages:

When an application is split or merged, the Client side needs to be aware (if you want to be unaware, you need to maintain a storage of the mapping relationship between the interface and the application); if you want to keep the original interface dimension of Dubbo for users, you need a lot of work to ensure; to reduce the transparency to users, you need to provide some other tools on OPS. For example, application developers can check whether a specific IP provides a service, and so on. Dubbo and Spring Cloud

Goal:

The service discovery of Dubbo and Spring Cloud is unified; Dubbo and Spring Cloud can call each other. Unified service discovery

Dubbo is transformed into service registration in the application dimension. (do not expand specifically, as explained in the following article)

Get through the call

In the Dubbo implementation, the support will be exposed as the Rest protocol and identified by Spring Cloud.

Dubbo + K8s

As explained in K8s, the following content also assumes that an application is deployed in a container and a container is deployed in a pod.

The following discussion of the scheme is actually related to each other, for example, service governance may affect service registration discovery, and service query cannot rely on the content of service registration. The whole design process is a process of continuous optimization. What is said below is exemplified by Dubbo.

Service governance

The service governance in the original system of Dubbo is strongly dependent on IP. When a set of service governance rules are configured, they are all based on one or more IP addresses.

After entering the K8s system, it is important to consider that the IP of Pod is not fixed. Therefore, the current routing rules can not meet the conditions, and will generate a lot of rule junk data. Under the K8s system, finding Pod through service is based on label selector, while managing Pod through deployment is also based on Pod label selector. So pod label selector is a general solution in K8s exercises.

Take routing rules as an example, a new routing rule, label routing, needs to be supported. After certain conditions are matched, the result is located in the Pod list queried by label selector instead of the original ip list.

To support label routing, the client side needs to obtain its own Pod label information on the client side, and it also needs to obtain the label information of each Pod in the server pod list.

Apply the way to get the information of current Pod

Pod defines environment variables and the application acquires them. Dubbo provides support for reading environment variables. Pod needs to set specific pod information according to the environment variables defined by Dubbo.

Pod information is transmitted through Downward API; Dubbo needs to provide directory reading to Downward, and the corresponding configuration of downward needs to be customized in Pod.

Get data through API Server; the most powerful way, but the application needs to be strongly dependent on API Server. Apply the way to obtain other Pod information

Get it by calling other Pod services; rely on the application to get its own Pod information, while exposing its own Pod information as a service (rest or dubbo protocol). The client side obtains the complete information of the corresponding Pod by calling the corresponding Pod.

Getting data through Api server; powerful, but with increased reliance on Api server. Service registration and discovery

Under the K8s system, RPC service discovery has the following ways:

Registration mechanism: write the IP to the registry and keep the connection with the heartbeat; when the heart stops, delete it from the registry

With Service+DNS: create a new Service, and you can select a set of Pod lists through the tag, which corresponds to an unchanged cluster IP; the Client side accesses the cluster IP via DNS or directly. This cluster IP is approximately equivalent to the implementation of load balancing (iptable mode)

Using headless service (DNS): the difference between headless service and the above service is that it does not provide cluster IP, but obtains a set of IP lists in the form of hostnames, and the Client side decides which Pod to access

Api server: the Client side directly requests api server to get the list of pod, and Client decides to access the logic of pod. At the same time, add watch when obtaining, and api server will synchronize the change information of pod with Client.

By getting the IP or host on the Server side, the Client side can initiate a request for http or other protocols.

Here are the possible scenarios that conform to Dubbo:

1. Dubbo + Zookeeper pod cluster (HSF+CS cluster)

This is the easiest way, Dubbo itself does not need to do any modification. The problem is that it increases the maintenance of Zookeeper, and this solution is not cloud-based and has nothing to do with K8s system.

Brain storm

The above solution is to use ZooKeeper as a registry, so can you use service in K8s as a registry? Each interface in Dubbo creates a service, updates the Endpoint information during the startup of each application instance, and establishes the relationship of Service- > Endpoint- > IP list.

In this scheme, the definition of K8s service has been modified, and there are too many service defined, so the maintenance and management of service is a difficult problem.

Scene based on K8s

In the traditional field of RPC, services are divided into service registration and service discovery. In the field of K8s, there is an one-to-one relationship between pod and applications. K8s itself provides the ability to find pod, so to a certain extent, service registration can not exist, but only service discovery. But this actually requires a premise:

Dubbo needs to transform the granularity of service registration discovery into an application dimension. > at the operation and maintenance level, write app=xxx (application name) to the label of pod.

2. Dubbo + K8s DNS

If cluster ip is provided by K8s service, then Dubbo is only responsible for calling the cluster Ip, and the logic of routing and load balancing is handed over to K8s proxy. This program reduces the core competencies of Dubbo. Let's move on to the capabilities provided by headless service.

By requesting.. svc.. IN An initiates a request to get the IP list, but it requires polling to get the updated IP list.

The functions related to service governance need to be supported independently in the above service governance section.

Reference: https://github.com/kubernetes/dns/blob/master/docs/specification.md#24---records-for-a-headless-service

3. Dubbo + Api Server

For the Dubbo application deployed in the container of Pod, the service registration process can be deleted directly, and the service discovery function can obtain Pod and service information through interaction with Api Server, as well as changes in watch pod and service. In this way, information related to service governance can also be obtained directly through Api Server.

4. Dubbo + Istio + Envoy

Dubbo can directly call the Envoy under the same pod (or the Envoy of the same node) by specifying the ip+ port. Dubbo gives routing rules, load balancing, circuit breakers and other functions to Istio and Envoy. Envoy needs to support forwarding of Dubbo protocol.

So Dubbo needs to accomplish two things: local IP direct connection (existing functionality) and redundant functionality tailoring (not implemented yet).

5. Dubbo + Istio

Dubbo applications no longer rely on Envoy as sidecar, but interact directly with Istio, using Istio as registry and configuration center for service governance; Dubbo needs to provide similar xDS protocol to convert service instance into dubbo protocol format in pilot; Dubbo also needs to adapt some functions of istio, such as health check and security-related logic. For specific implementation, you can refer to the implementation of Envoy. 6. There are many options for the coexistence of Dubbo and Istio under the K8s system. I would like to provide two ideas for you to think about:

All service registration is done through k8s mechanism, and all service discovery is done through Headless service. In the process of creating sidecar, you need to update the original K8s service.

Nacos serves as the registry of Dubbo and needs to partially transfer the data in K8s. For Dubbo applications, register the service to register the application dimension to Nacos, and Istio Pilot needs to identify Nacos data; the operating mechanism of Istio is basically unchanged, and the data of K8s service instance needs to be written to nacos for Dubbo to call. 7. Coexistence of above-cloud and under-cloud environment & multi-cluster environment on cloud

Istio provides cross-cluster and cloud-based solutions, and kubeFed as a cross-cluster solution for K8s can also play a role.

The complexity of this topic is higher, and I have some answers in mind. I hope you will also have some thoughts through the above.

Service query

Throw out three ways for everyone to think about.

Dubbo original mode

The original service query of Dubbo is for interface, and each interface will have a version number and group. The interface name + version number + group determines the unique service set, under which there are corresponding service providers and service consumers (interface-level dependencies). The service provider is a set of ip+port lists, and the service consumers are also a set of ip+port lists.

As an application-level service registration or directly using the Pod discovery mechanism that comes with K8s, some changes need to be made, which, as mentioned earlier, will be explained separately in other articles.

Only apply queries are supported

Similar to Spring Cloud, it supports queries that apply dimensions. After the specific application is queried, the ip+port list is included under the application details. Each ip+port is actually an instance of the application. Click on each application instance to view the details of each application, which includes the services provided by the instance.

API + Application query balance

On the basis of the original support for only application queries, add one step: support querying the list of applications corresponding to an interface, and most of the interfaces belong to only one application.

After clicking on the specific application under the application list, you will skip to the application details.

Summary

What is discussed above is an open source solution, so it is relatively less historical burden. For some large companies to switch from the original RPC scheme to cloud native support, they need to consider more compatibility and performance, and need to pay a higher price.

The cloud native trend is unstoppable, and it's too early to say which solution will win in RPC. I believe that both Service Mesh and the traditional RPC (Dubbo/ gRPC) will have their own place, so let time give us the answer.

"Alibaba Cloud's native Wechat official account (ID:Alicloudnative) focuses on micro-services, Serverless, containers, Service Mesh and other technology areas, focuses on cloud native popular technology trends, and large-scale cloud native landing practices, and is the technical official account that best understands cloud native developers."

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.