Shulou(Shulou.com)05/31 Report--

This article mainly introduces "how to use Storm". In daily operation, I believe many people have doubts about how to use Storm. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the doubts about "how to use Storm"! Next, please follow the editor to study!

Project Pom (Storm jar is not submitted to the Maven central warehouse, the following warehouse address needs to be added to the project):

Central Maven Repository Switchboard default http://maven.oschina.net/content/groups/public/ false clojars https://clojars.org/repo/ false true org.yaml snakeyaml 1.13 Org.apache.zookeeper zookeeper 3.3.3 org.clojure clojure 1.5.1 storm storm 0.9.0.1 storm libthrift7 0.7.0

The following is an example of Storm's HelloWord, the code has been deleted, readers familiar with Storm can naturally organize the code into a complete example.

Public static void main (String [] args) {Config conf = new Config (); conf.put (Config.STORM_LOCAL_DIR, "/ Volumes/Study/data/storm"); conf.put (Config.STORM_CLUSTER_MODE, "local"); / / conf.put ("storm.local.mode.zmq", "false"); conf.put ("storm.zookeeper.root", "/ storm") Conf.put ("storm.zookeeper.session.timeout", 50000); conf.put ("storm.zookeeper.servers", "nowledgedata-n15"); conf.put ("storm.zookeeper.port", 2181); / / conf.setDebug (true); / / conf.setNumWorkers (2); TopologyBuilder builder = new TopologyBuilder (); builder.setSpout ("words", new TestWordSpout (), 2) Builder.setBolt ("exclaim2", new DefaultStringBolt (), 5) .shuffleGrouping ("words"); LocalCluster cluster = new LocalCluster (); cluster.submitTopology ("test", conf, builder.createTopology ());}

Config.STORM_LOCAL_DIR is to configure a local path in which Storm writes some configuration information and temporary data.

Config.STORM_CLUSTER_MODE is the running mode, local and distributed are two options, that is, local mode and distributed mode. The local mode is multithreaded simulation at run time, which is used for development and testing; the distributed mode is multi-process and truly distributed under the distributed cluster.

Storm's Spout and Blot high availability are coordinated through ZooKeeper. Storm.zookeeper.root is a ZooKeeper address and has a corresponding port number.

Debug is in test mode and has more detailed log information.

TestWordSpout is an example that comes with Storm, which is used to randomly generate strings in new String [] {"nathan", "mike", "jackson", "golda", "bertels"}; lists to provide data sources.

The source code of DefaultStringBolt:

OutputCollector collector;public void prepare (Map conf, TopologyContext context, OutputCollector collector) {this.collector = collector;} public void execute (Tuple tuple) {log.info ("rev a message:" + tuple.getString (0)); collector.emit (tuple, new Values (tuple.getString (0) + "!!")); collector.ack (tuple);}

Run log:

10658 [Thread-29-exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: jackson 10658 [Thread-31-exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: jackson 10758 [Thread-26-exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: mike 10758 [Thread-33-exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: nathan 10859 [Thread-26-exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: nathan 10859 [Thread-29-exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: bertels 10961 [Thread-31-exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: jackson 10961 [Thread-33-exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: jackson 11061 [Thread-35-exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: nathan 11062 [Thread-35-exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: nathan 11162 [Thread-26-exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: bertels 11163 [Thread-26-exclaim2] INFO cn.pointways.dstorm.bolt.DefaultStringBolt-rev a message: jackson

The data is generated by a Storm called a nozzle (Spout, which is also the equivalent of a faucet, which produces the source side of the data), which is then passed to a series of Blot at the back end, which is eventually converted and consumed. Spout and Blot are both parallel, and the parallelism can be set on their own (the local run is simulated by multithreading). Such as:

Builder.setSpout ("words", new TestWordSpout (), 2); builder.setBolt ("exclaim2", new DefaultStringBolt (), 5)

The parallelism of nozzle TestWordSpout is 2. The parallelism of DefaultStringBolt is 5. 5.

As you can see from the log, the data goes through the nozzle to a predetermined Blot and the log is printed. The parallelism set by my test code is 5, and according to the statistics in the log, it is indeed 5 threads:

Thread-29-exclaim2

Thread-31-exclaim2

Thread-26-exclaim2

Thread-33-exclaim2

Thread-35-exclaim2

To borrow the words of OSC netizens, Hadoop is an automatic elevator in a shopping mall, users need to wait in line, choose the floor, and then arrive, while Storm is like an escalator, after the escalator is pre-set for operation, the incoming person will be transported away immediately, and the destination is clear.

Storm as I understand it, Storm and Hadoop are completely different, and there is no fitting part in the design. Storm is more like the Spring Integration I introduced earlier, which is a data flow system. It can transform, transmit, decompose and merge the data according to the preset process, and finally the data can be stored in the back end. It's just that Storm can be distributed, and the ability to distribute can be set by yourself.

This feature of Storm is very suitable for the development of ETL systems like big data.

At this point, the study on "how to use Storm" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.