Shulou(Shulou.com)05/31 Report--

This article mainly explains "how to use Storm DRPC". Friends who are interested might as well take a look. The method introduced in this paper is simple, fast and practical. Now let the editor take you to learn how to use Storm DRPC.

The introduction of DRPC into Storm mainly uses the real-time computing power of storm to parallelize CPU-intensive (CPU intensive) computing tasks. The stormtopology of DRPC takes the parameter stream of the function as input, and the return value of these function calls as the output stream of topology.

DRPC is not really a feature of storm itself, it is a pattern formed by combining storm primitives stream, spout, bolt, and topology. DRPC was supposed to be packed in a separate package, but DRPC was so useful that we bundled it with storm.

Overview

Distributed RPC is coordinated by a "DPRC server" (storm comes with an implementation). DRPC server coordination: ① receives a RPC request ② to send a request to storm topology ③ to receive the result from storm topology ④ to send the result back to the waiting client. From the client's point of view, a DRPC call is no different from a normal RPC call. For example, the following is the result of how the client invokes the RPC calculation "reach" function (function)

DRPCClient client = new DRPCClient ("drpc-host", 3772)

String result = client.execute ("reach", "http://twitter.com");"

The workflow for DRPC goes something like this (important ☆):

The client sends the name of the function (function) to be executed to the DRPC server, as well as the parameters of this function. The topology that implements this function uses DRPCSpout to receive a stream of function calls from the DRPC server, and each function call is marked with a unique id by the DRPC server. The topology then calculates the result, and at the end of the topology, a bolt called ReturnResults connects to the DRPC server and sends the result of this call to the DRPC server (through that unique id identity). The DRPC server uses that unique id to match the waiting client, wakes up the client and sends the result to it.

LinearDRPCTopologyBuilder

Storm comes with a topology builder called LinearDRPCTopologyBuilder, which automates almost all the steps to implement DRPC. These steps include:

1. Set spout

2. Return the result to the DRPC server

3. Provide bolt with limited ability to aggregate several groups of tuples.

Let's take a look at a simple example. Here is an example of adding a "!" after the input parameter. The implementation of DRPC topology based on

Public static class ExclaimBolt extends BaseBasicBolt {

Public void execute (Tuple tuple, BasicOutputCollector collector) {

String input = tuple.getString (1)

Collector.emit (new Values (tuple.getValue (0), input + "!"))

}

Public void declareOutputFields (OutputFieldsDeclarer declarer) {

Declarer.declare (new Fields ("id", "result"))

}

}

Public static void main (String [] args) throws Exception {

LinearDRPCTopologyBuilder builder = new LinearDRPCTopologyBuilder ("exclamation")

Builder.addBolt (new ExclaimBolt (), 3)

/ /...

}

It can be seen that there is very little we need to do. When you create a LinearDRPCTopologyBuilder, you need to tell it the name of the DRPC function (DRPC function) you want to implement. A DRPC server can coordinate many functions, and functions are distinguished by function names. The first bolt you declare will receive a two-dimensional tuple, the first field of tuple is request-id, and the second field is the parameters of the request. LinearDRPCTopologyBuilder also asks the last bolt of our topology to send a two-dimensional tuple like [id, result]: the first field is request-id, and the second field is the result of this function. Finally, the first field of all intermediate tuple must be request-id.

In this example, ExclaimBolt simply adds a "!" after the second field of tuple, and LinearDRPCTopologyBuilder does the rest for us: connect to the DRPC server and send the results back.

Local mode DRPC

DRPC can be run in local mode. Here is the code to run the above example in local mode:

LocalDRPC drpc = new LocalDRPC ()

LocalCluster cluster = new LocalCluster ()

Cluster.submitTopology ("drpc-demo", conf, builder.createLocalTopology (drpc))

System.out.println ("Results for 'hello':" + drpc.execute ("exclamation", "hello"))

Cluster.shutdown ()

Drpc.shutdown ()

First you create a LocalDRPC object that simulates a DRPC server in the process (much like LocalCluster simulates a Storm cluster in the process), and then creates a LocalCluster object to run topology in local mode. LinearTopologyBuilder has separate ways to create local topology and remote topology. In native mode, the LocalDRPC object is not bound to any port, so our topology object needs to know who to interact with, which is why the createLocalTopology method accepts a LocalDRPC object as input.

Once topology is started, you can call the RPC method by calling the execute of the LocalDRPC object.

Remote mode DRPC

DRPC is also very simple on a real cluster, with three steps:

1. Start the DRPC server

2. Configure the address of the DRPC server

3. Submit the DRPCtopology to the storm cluster.

We can start the DRPC server with the "bin/storm drpc" command.

Next, you need to let your storm cluster know the address of your DRPC server. DRPCSpout needs this address to receive function calls from the DRPC server. This can be configured in storm.yaml or in topology by code. The configuration via storm.yaml is as follows:

Drpc.servers:

-"drpc1.foo.com"

-"drpc2.foo.com"

Finally, you submit the DRPC topology through the StormSubmitter object (this is no different from the other topology you submit). To run the above example remotely, use the following code:

StormSubmitter.submitTopology ("exclamation-drpc", conf, builder.createRemoteTopology ())

We use the createRemoteTopology method to create a DRPC topology that runs on a real cluster.

At this point, I believe you have a deeper understanding of "how to use Storm DRPC". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.