Shulou(Shulou.com)05/31 Report--

This article introduces how to analyze the implementation mechanism of Storm DRPC. The content is very detailed. Interested friends can use it for reference. I hope it will be helpful to you.

DRPC is a high-level abstraction based on the basic concepts of Storm (Topology, Spout, Bolt, Stream, etc.). It is understood that its goal is to provide a distributed RPC framework on top of the Storm cluster, so that it can quickly realize the distributed computing process of RPC requests using Storm, that is, initiating a RPC request, multiple worker computing nodes and computing, and finally summarizing and returning the calculation results to the client.

Thrift is used as its RPC framework in Storm. Similarly, the implementation of DRPC is also built on the Thrift protocol, and the related source files are as follows:

1. Storm-core/src/storm.thrift, which defines the Thrift protocol implemented in Storm, in which two service are related to DRPC: DistributedRPC and DistributedRPCInvocations. Their interfaces are defined as follows:

DistributedRPC.Iface: defines the execute method, which is used by the client to initiate RPC requests

DistributedRPCInvocations.Iface: define fetchRequest, failRequest, and result methods, which are used to obtain the RPC request, mark the RPC request as failed, and return the processing result of the RPC request.

2. Storm-core/src/clj/backtype/storm/daemon/drpc.clj, which implements the Thrift server of DRPC (that is, DRPC Server), using Clojure language.

3. Storm-core/src/jvm/backtype/storm/generated/DistributedRPC.java and storm-core/src/jvm/backtype/storm/utils/DRPCClient.java, as RPC clients, implement DistributedRPC.Iface interface, which is used for clients to initiate RPC requests to DRPC Server.

4. Storm-core/src/jvm/backtype/storm/generated / DistributedRPCInvocations.java and storm-core/src/jvm/backtype/storm/drpc / DRPCInvocationsClient.java, as RPC clients, implement the DistributedRPCInvocations.Iface interface, which is used for DRPC Topology to trigger the execution of DRPC Request and return the result to DRPC Server.

As you can see, for DRPC Server, DRPC Client and DRPC Topology are both clients of Thrift, just calling different Thrift services.

Storm DRPC implementation architecture

1. First, the premise is that DRPC Topology is already running on the cluster, and each DRPC service registers a RPC method, including the method name and parameter form (the figure above assumes that Topology is up and running)

two。 Next is the processing flow. The client calls the execute method through DRPCClient and initiates a RPC call to DRPC Server. Currently, only one DRPC method call parameter of type String is supported, which is being discussed in the community.

3. Then, there is a handler-server pool in DRPC Server that receives RPC requests, generates a unique request id for each request, generates a DRPC Request record, and puts it in request queue waiting to be consumed (calculated)

4. Finally, the relevant modules in DRPC Topology (DRPC Spout, ReturnResults Bolt, described later) fetch the RPC request for this method from request queue through invoke-server pool and return the processing result (success / failure) to DRPC Server until it is finally returned to the blocked DRPC Client.

Storm DRPC topology data flow

The DRPCTopology consists of 1 DRPCSpout, 1 Prepare-Request Bolt, several User Bolts (that is, the Bolts added by the user through LinearDRPCTopologyBuilder), 1 JoinResult Bolt and 1 ReturnResults Bolt. Except for User Bolts, everything else is added to Topology by LinearDRPCTopologyBuilder built-in. Next, let's look at how these Spout and Bolts work from the flow relationship of the data flow:

1. Several DRPCInvocationsClient are maintained in DRPCSpout. The RPC request that needs to be submitted to Topology is read from DRPC Server by fetchRequest method, and then a data stream is sent to Prepare-Request Bolt:

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.