Shulou(Shulou.com)06/02 Report--

Author | Liu Jun (Land Turtle) Apache Dubbo PMC

Overview

Starting from version 2.7.5 of Dubbo, the community version introduces a new service discovery mechanism based on instance (application) granularity, which is an important step in our exploration of adapting cloud native infrastructure for Dubbo. It has been nearly half a year since the release of the version. after exploring and summarizing this period of time, we have a more comprehensive and in-depth understanding of the feasibility and stability of this mechanism. At the same time, the planning of Dubbo 3.0 is also in full progress, how to make application-level service discovery become the basic service model of the future next-generation service framework Dubbo 3.0, and how to solve the problems of cloud native and large-scale micro-service cluster expansion and scalability has also become the focus of our current work.

Since this new mechanism is so important, how does it work? Today we will interpret it in detail. In the original community version, we gave this mechanism a mysterious name-service introspection. The following article will further explain the origin of this name and refer to the application-level service discovery mechanism.

Developers who are familiar with Dubbo should know that services have always been defined for RPC methods, and this is the basis for Dubbo development friendliness and governance capabilities. In that case, why should we define a service discovery mechanism with application granularity? How exactly does this mechanism work? What is the difference between it and the current mechanism? What benefits can it bring to us? How does it help to adapt to cloud natives and improve performance?

With all these questions, we begin to explain this article.

What is service introspection?

First of all, let's explain the problems mentioned at the beginning of the article:

What kind of model is application granularity service discovery, and what is the difference between it and the current Dubbo service discovery model?

Why do we call it serving introspection?

The so-called "application / instance granularity" or "RPC service granularity" emphasizes a data organization format for address discovery.

Take Dubbo's current address discovery data format as an example, which is "RPC service granularity". It organizes the data with RPC service as key and instance list as value:

"RPC Service1": [

{"name": "instance1", "ip": "127.0.0.1", "metadata": {"timeout": 1000}}

{"name": "instance2", "ip": "127.0.0.1", "metadata": {"timeout": 2000}}

{"name": "instance3", "ip": "127.0.0.1", "metadata": {"timeout": 3000}}

]

"RPC Service2": [Instance list of RPC Service2]

"RPC ServiceN": [Instance list of RPC ServiceN]

Our newly introduced "Application granularity Service Discovery" takes the application name (Application) as the key and the list of instances (Instance) deployed by the application as the value. This makes two differences:

The data mapping relationship has changed from RPC Service-> Instance to Application-> Instance

There is less data, and the registry does not have RPC Service and its related configuration information.

"application1": [

{"name": "instance1", "ip": "127.0.0.1", "metadata": {}}

{"name": "instance2", "ip": "127.0.0.1", "metadata": {}}

{"name": "instanceN", "ip": "127.0.0.1", "metadata": {}}

]

To further understand the changes brought about by the new model, let's take a look at the relationship between applications and RPC services. It is obvious that n RPC Service may be defined within an application. Therefore, the granularity of service discovery before Dubbo is finer, and there will be more data entries in the registry (proportional to RPC services). At the same time, there is also some data redundancy.

Briefly understand the basic mechanism of application-level service discovery, and then explain why it is called "service introspection".

In fact, we have to start from how it works. As mentioned above, there are several obvious changes in the data model discovered by the application granularity service: the amount of data in the data center is less, the data related to RPC services is gone in the registry, and now there are only two levels of data: application-instance. In order to ensure that the missing RPC service data can still be correctly perceived by the Consumer, we have established a separate communication channel between Consumer and Provider: Consumer and Provider exchange information through a specific port. We regard the behavior of Provider actively exposing its own information as a kind of introspection mechanism, so from this point of view, we name the whole mechanism: service introspection.

Why do you need service introspection?

When talking about the general principle of service introspection, I also mentioned several differences it brings to the registry. These differences are reflected in the Dubbo framework (and even in the whole micro-service system), which have the following advantages:

Align with the mainstream microservice models in the industry, such as SpringCloud, Kubernetes Native Service, etc.

Improve performance and scalability. The reorganization (reduction) of registry data can greatly reduce the storage and push pressure on the registry, thereby reducing the address calculation pressure on the Dubbo Consumer side, and the cluster size has become predictable and evaluable (independent of the number of RPC interfaces, but only related to the deployment scale of the instance).

1. Align mainstream microservice models

Automatic and transparent instance address discovery (load balancing) is something that all micro-service frameworks need to solve, which makes the deployment structure of the back-end transparent to the upstream micro-service, which only needs to select one of the received address lists and initiate the call. To achieve the above goals, automatic synchronization of two key points is involved:

The address of the instance, which the service consumer needs to know to establish a link

RPC method definition, the service consumer needs to know the specific definition of the RPC service, regardless of whether the service type is rest or rmi, and so on.

For data synchronization between RPC instances with the help of registry, REST defines a very interesting maturity model. Interested friends can refer to this link:

Https://www.martinfowler.com/articles/richardsonMaturityModel.html

According to the level 4 maturity definition in this article, Dubbo's current model based on interface granularity can correspond to the L4 level.

Next, let's take a look at how Dubbo, SpringCloud, and Kubernetes are designed around the goal of automated instance address discovery.

2. Spring Cloud

SpringCloud synchronizes only the application and instance addresses through the registry, and the consumer can establish a link with the service provider based on the instance address, but the consumer knows nothing about how to initiate HTTP calls (SpringCloud communicates based on rest), such as what HTTP endpoint the other party has and which parameters need to be passed in.

This part of the RPC service information is currently negotiated through offline agreements or offline management systems. The advantages and disadvantages of this architecture are summarized as follows:

Advantages: clear deployment structure and small number of address push

Disadvantages: address subscription needs to specify application name, provider application change (split) needs consumer awareness; RPC calls cannot be synchronized automatically.

3. Dubbo

Dubbo synchronizes the instance address and the RPC method through the registry, so it can automatically synchronize the RPC process, facing RPC programming and RPC governance, and is not aware of the split consumer end of the back-end application. The disadvantage is that the number of address pushes increases, which is proportional to the RPC method.

4. Dubbo + Kubernetes

Dubbo needs to support Kubernetes native service. Compared with the previous self-built registry service discovery system, there are two main changes in the working mechanism:

Service registration is taken over by the platform, and provider no longer needs to care about service registration

Service discovery at the consumer end will be the focus of Dubbo. By interfacing with API-Server and DNS at the platform layer, Dubbo client can query a group of Endpoints (a group of pod running provider) through a Service Name (usually corresponding to Application Name), and map the Endpoints to the internal address list of Dubbo to drive the work of Dubbo's built-in load balancing mechanism.

As an abstract concept, how to map Kubernetes Service to Dubbo is a point worth discussing.

Service Name-> Application Name,Dubbo applications correspond to Kubernetes services one by one, transparent for microservice operation and construction, and decoupled from the development phase.

ApiVersion: v1

Kind: Service

Metadata:

Name: provider-app-name

Spec:

Selector:

App: provider-app-name

Ports:

-protocol: TCP

Port:

TargetPort: 9376

Service Name-> Dubbo RPC Service,Kubernetes the services to be maintained and scheduled are bound to the application's built-in RPC services, and the number of services maintained becomes larger.

-

ApiVersion: v1

Kind: Service

Metadata:

Name: rpc-service-1

Spec:

Selector:

App: provider-app-name

Ports: # #

...

-

ApiVersion: v1

Kind: Service

Metadata:

Name: rpc-service-2

Spec:

Selector:

App: provider-app-name

Ports: # #

...

-

ApiVersion: v1

Kind: Service

Metadata:

Name: rpc-service-N

Spec:

Selector:

App: provider-app-name

Ports: # #

...

Combined with the analysis of the above different micro-service framework models, we can find that there are great differences in the abstract definition of micro-services between Dubbo and different products such as SpringCloud and Kubernetes. SpringCloud and Kubernetes are quite similar in the model abstraction of microservices. Basically, both of them are only concerned with the synchronization of instance addresses. If we look at some other service framework products, we will find that most of them are designed in the same way.

This is the L3 level in the REST maturity model.

In contrast, Dubbo is relatively special, and it is more designed from the granularity of RPC services.

Corresponds to the L4 level in the REST maturity model.

For example, we have made a detailed analysis of each model above, and each model has its own advantages and disadvantages. And we initially decided that Dubbo should make a change to discover the alignment on the model of other microservices, which was the first time when we determined the cloud native solution of Dubbo, we found that model alignment was a basic condition for Dubbo to support Kubernetes Native Service. Another point is the demand for some engineering practices of Dubbo scene from the user side. Thanks to Dubbo's support for multi-registration and multi-protocol capabilities, it is possible for Dubbo to communicate with different micro-service systems, and the inconsistency of service discovery model has become one of the obstacles. For the scenario description of this part, please see here.

5. Larger micro-service clusters-solving performance bottlenecks

This part involves the interaction with the registry and configuration center, and we have briefly analyzed the changes of registry data under different models. For a more intuitive comparison of the push efficiency improvement brought about by service model changes, let's take a look at the comparison of different model registries through an example:

On the left side of the figure is a typical workflow of the microservices framework, where Provider and Consumer automate address notification through a registry. The information of Provider instance is shown in the table in the figure:

The application DEMO includes three interfaces DemoService 1 2 3, and the ip address of the current instance is 10.210.134.30.

For Spring Cloud and Kubernetes models, the registry stores only one piece of DEMO-10.210.134.30+metadata data

For the old Dubbo model, the registry stores three interface granularity data, corresponding to three interfaces DemoService 1 / 2 / 3, and a lot of address data are duplicated.

It can be concluded that the amount of data stored and pushed by the model based on application granularity is proportional to the number of applications and instances, and the address push pressure will increase only when the number of applications increases or the number of instances increases.

For the model based on interface granularity, the amount of data is positively related to the number of interfaces. In view of the current situation that an application usually publishes multiple interfaces, this order of magnitude itself is multiplied by the application granularity. Another key point is that the opacity of cluster size evaluation caused by interface granularity, compared with real examples and application growth, is usually in the planning of the operation and maintenance side, and the definition of the interface is more about the internal behavior of the business side. it is often possible to bypass the evaluation and put pressure on the cluster.

Take the Consumer service subscription as an example. According to my rough statistics of the large-scale head users in some Dubbo in the community, according to the actual scenario of the statistical company, the number of Provier applications consumed (subscribed) by a Consumer application is often more than 10, while the specific number of interfaces to be consumed (subscribed) is usually up to 30. On average, the 3 interfaces subscribed to by Consumer come from the same Provider application. If the application granularity is used as the basic unit of address notification and address selection, the average address push and computation will be reduced by more than 60%.

In extreme cases, when more of the interfaces consumed by the Consumer end come from the same application, the usage of this address push and memory consumption will be further reduced, even more than 80%.

A typical scenario is gateway-based applications in the Dubbo architecture. Some gateway applications consume (subscribe) 100 + applications, while consuming (subscribing) services have 1000 +. On average, 10 interfaces come from the same application. If we change the granularity of address push and computing to applications, the number of address pushes will be changed from n 100 to n 1000, and the number of addresses can be reduced by nearly 90%.

working principle

1. Design principle

In the previous section, we gave the benefits or reasons why Dubbo is closer to application-level service discovery from the point of view of service model and supporting large-scale clusters, but at the same time, the service governance capability of interface granularity should be retained, which is the basis of the ease of use of Dubbo framework programming model and service governance ability.

The following are the design principles that I think we should still adhere to when we do service model migration:

The new service discovery model is to realize the imperceptible migration of the original Dubbo consumer developers, that is, Dubbo continues to program for RPC services and RPC services governance, so that it is completely unaware of the user side.

The automatic RPC service metadata coordination mechanism between Consumer and Provider is established to solve the disadvantage that the traditional micro-service model can not synchronize the RPC-level interface configuration.

two。 Detailed explanation of the basic principles

As a new service discovery mechanism, application-level service discovery is basically consistent with the previous Dubbo service discovery based on RPC service granularity in the core process, that is, service providers register address information with the registry, and service consumers pull and subscribe address information from the registry.

The main differences here are as follows:

Registry data is organized in the format "Application-instance list" and no longer contains RPC service information

The following is sample data for each Instance metadata. The general principle is that metadata only contains information related to the current instance node and does not involve information about RPC service granularity.

The overall information is summarized as follows: instance address, various environment labels of the instance, metadata service metadata, and a small number of other necessary attributes.

{

"name": "provider-app-name"

"id": "192.168.0.102 purl 20880"

"address": "192.168.0.102"

"port": 20880

"sslPort": null

"payload": {

"id": null

"name": "provider-app-name"

"metadata": {

"metadataService": "{\" dubbo\ ": {\" version\ ":\" 1.0.0\ ",\" dubbo\ ":\" 2.0.2\ ",\" release\ ":\" 2.7.5\ ",\" port\ ":\" 20881\ "}

"endpoints": "[{\" port\ ": 20880,\" protocol\ ":\" dubbo\ "}]"

"storage-type": "local"

"revision": "6785535733750099598"

}

}

"registrationTimeUTC": 1583461240877

"serviceType": "DYNAMIC"

"uriSpec": null

}

Client-Server negotiates the RPC method information by itself.

After the registry no longer synchronizes RPC service information, service introspection establishes a built-in RPC service information negotiation mechanism between the service consumer and the provider, which is the origin of the name "service introspection". The server-side instance exposes a predefined MetadataService RPC service, and the consumer gets the configuration information related to each instance's RPC method by calling MetadataService.

The data returned by MetadataService is in the following format:

[

"dubbo://192.168.0.102:20880/org.apache.dubbo.demo.DemoService?anyhost=true&application=demo-provider&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&interface=org.apache.dubbo.demo.DemoService&methods=sayHello&pid=9585&release=2.7.5&side=provider×tamp=1583469714314"

"dubbo://192.168.0.102:20880/org.apache.dubbo.demo.HelloService?anyhost=true&application=demo-provider&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&interface=org.apache.dubbo.demo.DemoService&methods=sayHello&pid=9585&release=2.7.5&side=provider×tamp=1583469714314"

"dubbo://192.168.0.102:20880/org.apache.dubbo.demo.WorldService?anyhost=true&application=demo-provider&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&interface=org.apache.dubbo.demo.DemoService&methods=sayHello&pid=9585&release=2.7.5&side=provider×tamp=1583469714314"

]

Developers familiar with Dubbo's service discovery model based on RPC service granularity should be able to see that the service introspection mechanism splits the URL previously passed by the registry into two:

Some of the data related to the instance will remain in the registry, such as ip, port, machine identification, etc.

Another part of the data related to the RPC method is removed from the registry and is instead exposed to the consumer side through MetadataService.

Ideally, data can be strictly distinguished according to instances and RPC services, but it is obvious that there are still some data redundancy in the above implementation versions, and some data has not been reasonably divided. Especially in the MetadataService part, the data returned is just a simple URL list assembly, and these URL actually contain all the data.

The following is a complete workflow diagram of service introspection, which describes in detail the collaboration process among service registration, service discovery, MetadataService, and RPC invocations.

The service provider starts, first parses the "normal service" defined by the application and registers it as a RPC service in turn, then registers the built-in MetadataService service, and finally opens the TCP listening port

After the startup is completed, register the instance information with the registry (only ip, port and other instance-related data), and the provider launches

Service consumers start, first query the address list from the registry according to the "consumed provider application name", and complete the subscription (to achieve automatic notification of subsequent address changes)

After the consumer gets the address list, it immediately initiates a call to MetadataService, and the result contains all the "normal services" defined by the application and its related configuration information.

At this point, the consumer can receive external traffic and make Dubbo RPC calls to the provider.

In the above process, we only consider that everything is going well, but in a more detailed design or coding implementation, we also need to strictly agree on the framework behavior in some exception scenarios. For example, if the consumer MetadataService call fails, the consumer will not be able to receive external traffic until the retry knows it is successful.

The key Mechanism in Service introspection

1. Metadata synchronization mechanism

The configuration synchronization between Client and Server after receiving the address push is the key link of service introspection. At present, there are two specific options for metadata synchronization: build an independent metadata center for MetadataService;, and coordinate data through detailed metadata clusters.

The built-in MetadataService:MetadataService is exposed through the standard Dubbo protocol, and the qualified "normal service" configuration in memory is returned to the consumer according to the query conditions. This step occurs before the location and invocation of the consumer side

Metadata center: reuse the metadata center introduced in version 2.7. after the provider instance is launched, it will try to organize the internal RPC services into a metadata format to the metadata center, while consumer will actively query the metadata center after receiving an update pushed by the registry.

Note that the time for the consumer side to query the metadata center is to wait until the address update notification of the registry. That is, through the data issued by the registry, we can clearly know when the metadata of an instance has been updated, and then we need to check the metadata center.

2. RPC service

< - >

Apply mapping relationship

Reviewing the data organization form of the "application-instance list" structure in the registry mentioned above, this change is not completely transparent to developers, and the business development side will be aware of the change in the mechanism for querying / subscribing to address lists. Specifically, compared with the past when we retrieve addresses based on RPC services, consumer now needs to specify the provider application name in order to query or subscribe to addresses.

Examples of old Consumer development and configuration:

Examples of new Consumer development and configuration:

The above method of specifying provider application names is the current practice of Spring Cloud, which requires developers on the consumer side to display and specify the provider applications they want to consume.

The root cause of the above problem is that the registry does not know any information related to RPC services, so it can only be queried by the application name.

In order to make the whole development process more transparent to old Dubbo users and avoid the impact of specifying provider on scalability (see below), we designed a set of RPC service-to-application name mapping to try to automate the conversion of RPC services to provider application names in consumer.

The reason why Dubbo chooses to establish a set of "interface-application" mapping relationship is mainly considering the uncertainty of service-app mapping relationship. A typical scenario is application / service split. As the configuration mentioned above, PC Service 2 is a service defined in provider-app-x. In the future, it may be split into another new application such as provider-app-x-1 by developers at any time. This split will be perceived by all PC Service 2 consumers, and the application will be modified and upgraded. If changed, the upgrade cost is undeniably high.

Whether the Dubbo framework helps developers solve this problem transparently or leave it to developers themselves, of course, this is just a matter of strategy choice, and Dubbo version 2.7.5 + is currently available. In fact, I personally prefer to leave it to business developers to do through organizational constraints, which can further reduce the complexity of the Dubbo framework and improve the stability of the running state.

Summary and prospect

Application-level service discovery mechanism is an important step for Dubbo to face the cloud natively. It helps Dubbo bridge the gap with other micro-service systems in the level of address discovery, and becomes the basis for Dubbo to adapt to infrastructure such as Kubernetes Native Service.

We expect Dubbo to continue to retain its strong advantages in terms of programming ease of use, service governance capabilities, etc., based on the new model. But we should also see that on the one hand, the model of application granularity brings new complexity, which requires us to continue to optimize and enhance; on the other hand, in addition to address storage and push, application granularity also has the potential to help Dubbo location.

A brief introduction to the author

Liu Jun, Github account Chickenlj,Apache Dubbo PMC, the core development of the project, witnessed the whole process of Dubbo from restarting open source to graduating from Apache. Currently, he serves as the native application platform team of Ali Yunyun, and participates in the work related to service framework and micro services. Currently, he is mainly promoting the Dubbo 3.0-Dubbo cloud native.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.